2023-02-21 09:56:54,117 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f 2023-02-21 09:56:54,130 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-21 09:56:54,144 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-21 09:56:54,144 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88, deleteOnExit=true 2023-02-21 09:56:54,144 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-21 09:56:54,145 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/test.cache.data in system properties and HBase conf 2023-02-21 09:56:54,145 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/hadoop.tmp.dir in system properties and HBase conf 2023-02-21 09:56:54,146 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/hadoop.log.dir in system properties and HBase conf 2023-02-21 09:56:54,146 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-21 09:56:54,147 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-21 09:56:54,147 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-21 09:56:54,270 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-21 09:56:54,684 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-21 09:56:54,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-21 09:56:54,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-21 09:56:54,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-21 09:56:54,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-21 09:56:54,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-21 09:56:54,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-21 09:56:54,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-21 09:56:54,692 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-21 09:56:54,692 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-21 09:56:54,692 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/nfs.dump.dir in system properties and HBase conf 2023-02-21 09:56:54,692 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/java.io.tmpdir in system properties and HBase conf 2023-02-21 09:56:54,693 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-21 09:56:54,693 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-21 09:56:54,693 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-21 09:56:55,196 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-21 09:56:55,200 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-21 09:56:55,465 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-21 09:56:55,630 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-21 09:56:55,643 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-21 09:56:55,679 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-21 09:56:55,711 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/java.io.tmpdir/Jetty_localhost_37933_hdfs____.iezyld/webapp 2023-02-21 09:56:55,840 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37933 2023-02-21 09:56:55,877 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-21 09:56:55,877 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-21 09:56:56,291 WARN [Listener at localhost/35503] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-21 09:56:56,398 WARN [Listener at localhost/35503] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-21 09:56:56,415 WARN [Listener at localhost/35503] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-21 09:56:56,421 INFO [Listener at localhost/35503] log.Slf4jLog(67): jetty-6.1.26 2023-02-21 09:56:56,425 INFO [Listener at localhost/35503] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/java.io.tmpdir/Jetty_localhost_43857_datanode____.uke2e6/webapp 2023-02-21 09:56:56,519 INFO [Listener at localhost/35503] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43857 2023-02-21 09:56:56,821 WARN [Listener at localhost/41469] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-21 09:56:56,831 WARN [Listener at localhost/41469] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-21 09:56:56,834 WARN [Listener at localhost/41469] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-21 09:56:56,837 INFO [Listener at localhost/41469] log.Slf4jLog(67): jetty-6.1.26 2023-02-21 09:56:56,843 INFO [Listener at localhost/41469] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/java.io.tmpdir/Jetty_localhost_36467_datanode____qq49sp/webapp 2023-02-21 09:56:56,940 INFO [Listener at localhost/41469] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36467 2023-02-21 09:56:56,950 WARN [Listener at localhost/34733] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-21 09:56:56,970 WARN [Listener at localhost/34733] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-21 09:56:56,976 WARN [Listener at localhost/34733] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-21 09:56:56,977 INFO [Listener at localhost/34733] log.Slf4jLog(67): jetty-6.1.26 2023-02-21 09:56:56,982 INFO [Listener at localhost/34733] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/java.io.tmpdir/Jetty_localhost_46149_datanode____.io7cdh/webapp 2023-02-21 09:56:57,090 INFO [Listener at localhost/34733] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46149 2023-02-21 09:56:57,098 WARN [Listener at localhost/40243] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-21 09:56:57,279 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x80311bea67e396a7: Processing first storage report for DS-3c068c46-ef90-4903-925d-b2597b1e20eb from datanode d9b45635-f12f-447d-8966-6f56b17879d6 2023-02-21 09:56:57,280 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x80311bea67e396a7: from storage DS-3c068c46-ef90-4903-925d-b2597b1e20eb node DatanodeRegistration(127.0.0.1:39513, datanodeUuid=d9b45635-f12f-447d-8966-6f56b17879d6, infoPort=43631, infoSecurePort=0, ipcPort=40243, storageInfo=lv=-57;cid=testClusterID;nsid=1432038234;c=1676973415272), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-21 09:56:57,280 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xac1d866faa27f6f1: Processing first storage report for DS-8931b049-6053-42e9-9bd0-e03684384fda from datanode a8510abe-d1ee-4d75-bfe9-cb5544285d6e 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xac1d866faa27f6f1: from storage DS-8931b049-6053-42e9-9bd0-e03684384fda node DatanodeRegistration(127.0.0.1:44327, datanodeUuid=a8510abe-d1ee-4d75-bfe9-cb5544285d6e, infoPort=40057, infoSecurePort=0, ipcPort=34733, storageInfo=lv=-57;cid=testClusterID;nsid=1432038234;c=1676973415272), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7cc49cb4bed1ec55: Processing first storage report for DS-052ab17e-09a8-4f09-8081-ac55350de361 from datanode a7765cd4-64e4-40e6-b68d-4e6de46e2658 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7cc49cb4bed1ec55: from storage DS-052ab17e-09a8-4f09-8081-ac55350de361 node DatanodeRegistration(127.0.0.1:33051, datanodeUuid=a7765cd4-64e4-40e6-b68d-4e6de46e2658, infoPort=37239, infoSecurePort=0, ipcPort=41469, storageInfo=lv=-57;cid=testClusterID;nsid=1432038234;c=1676973415272), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x80311bea67e396a7: Processing first storage report for DS-b781f857-827e-4635-ae6f-f264d4163f93 from datanode d9b45635-f12f-447d-8966-6f56b17879d6 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x80311bea67e396a7: from storage DS-b781f857-827e-4635-ae6f-f264d4163f93 node DatanodeRegistration(127.0.0.1:39513, datanodeUuid=d9b45635-f12f-447d-8966-6f56b17879d6, infoPort=43631, infoSecurePort=0, ipcPort=40243, storageInfo=lv=-57;cid=testClusterID;nsid=1432038234;c=1676973415272), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xac1d866faa27f6f1: Processing first storage report for DS-a3186b21-f26a-4a73-9fa4-f8db0500b317 from datanode a8510abe-d1ee-4d75-bfe9-cb5544285d6e 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xac1d866faa27f6f1: from storage DS-a3186b21-f26a-4a73-9fa4-f8db0500b317 node DatanodeRegistration(127.0.0.1:44327, datanodeUuid=a8510abe-d1ee-4d75-bfe9-cb5544285d6e, infoPort=40057, infoSecurePort=0, ipcPort=34733, storageInfo=lv=-57;cid=testClusterID;nsid=1432038234;c=1676973415272), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7cc49cb4bed1ec55: Processing first storage report for DS-25ce3d27-e714-410d-a49c-0857274f0605 from datanode a7765cd4-64e4-40e6-b68d-4e6de46e2658 2023-02-21 09:56:57,281 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7cc49cb4bed1ec55: from storage DS-25ce3d27-e714-410d-a49c-0857274f0605 node DatanodeRegistration(127.0.0.1:33051, datanodeUuid=a7765cd4-64e4-40e6-b68d-4e6de46e2658, infoPort=37239, infoSecurePort=0, ipcPort=41469, storageInfo=lv=-57;cid=testClusterID;nsid=1432038234;c=1676973415272), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-21 09:56:57,492 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f 2023-02-21 09:56:57,552 INFO [Listener at localhost/40243] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/zookeeper_0, clientPort=58856, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-21 09:56:57,565 INFO [Listener at localhost/40243] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58856 2023-02-21 09:56:57,572 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:57,574 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:58,240 INFO [Listener at localhost/40243] util.FSUtils(479): Created version file at hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926 with version=8 2023-02-21 09:56:58,240 INFO [Listener at localhost/40243] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/hbase-staging 2023-02-21 09:56:58,539 INFO [Listener at localhost/40243] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-21 09:56:59,002 INFO [Listener at localhost/40243] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-21 09:56:59,034 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,034 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,035 INFO [Listener at localhost/40243] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-21 09:56:59,035 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,035 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-21 09:56:59,191 INFO [Listener at localhost/40243] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-21 09:56:59,264 DEBUG [Listener at localhost/40243] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-21 09:56:59,357 INFO [Listener at localhost/40243] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41687 2023-02-21 09:56:59,367 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,369 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,388 INFO [Listener at localhost/40243] zookeeper.RecoverableZooKeeper(93): Process identifier=master:41687 connecting to ZooKeeper ensemble=127.0.0.1:58856 2023-02-21 09:56:59,429 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:416870x0, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-21 09:56:59,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:41687-0x100764f8e080000 connected 2023-02-21 09:56:59,456 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-21 09:56:59,457 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:56:59,460 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-21 09:56:59,467 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41687 2023-02-21 09:56:59,468 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41687 2023-02-21 09:56:59,468 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41687 2023-02-21 09:56:59,468 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41687 2023-02-21 09:56:59,469 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41687 2023-02-21 09:56:59,474 INFO [Listener at localhost/40243] master.HMaster(444): hbase.rootdir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926, hbase.cluster.distributed=false 2023-02-21 09:56:59,540 INFO [Listener at localhost/40243] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-21 09:56:59,540 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,540 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,540 INFO [Listener at localhost/40243] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-21 09:56:59,541 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,541 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-21 09:56:59,545 INFO [Listener at localhost/40243] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-21 09:56:59,548 INFO [Listener at localhost/40243] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46077 2023-02-21 09:56:59,550 INFO [Listener at localhost/40243] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-21 09:56:59,556 INFO [Listener at localhost/40243] mob.MobFileCache(124): MobFileCache disabled 2023-02-21 09:56:59,557 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,558 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,560 INFO [Listener at localhost/40243] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46077 connecting to ZooKeeper ensemble=127.0.0.1:58856 2023-02-21 09:56:59,567 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:460770x0, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-21 09:56:59,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46077-0x100764f8e080001 connected 2023-02-21 09:56:59,568 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-21 09:56:59,569 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:56:59,570 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-21 09:56:59,571 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46077 2023-02-21 09:56:59,571 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46077 2023-02-21 09:56:59,571 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46077 2023-02-21 09:56:59,572 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46077 2023-02-21 09:56:59,572 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46077 2023-02-21 09:56:59,584 INFO [Listener at localhost/40243] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-21 09:56:59,584 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,584 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,585 INFO [Listener at localhost/40243] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-21 09:56:59,585 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,585 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-21 09:56:59,585 INFO [Listener at localhost/40243] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-21 09:56:59,588 INFO [Listener at localhost/40243] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44835 2023-02-21 09:56:59,588 INFO [Listener at localhost/40243] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-21 09:56:59,589 INFO [Listener at localhost/40243] mob.MobFileCache(124): MobFileCache disabled 2023-02-21 09:56:59,590 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,592 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,594 INFO [Listener at localhost/40243] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44835 connecting to ZooKeeper ensemble=127.0.0.1:58856 2023-02-21 09:56:59,598 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:448350x0, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-21 09:56:59,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44835-0x100764f8e080002 connected 2023-02-21 09:56:59,599 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-21 09:56:59,600 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:56:59,600 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-21 09:56:59,601 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44835 2023-02-21 09:56:59,601 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44835 2023-02-21 09:56:59,602 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44835 2023-02-21 09:56:59,603 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44835 2023-02-21 09:56:59,603 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44835 2023-02-21 09:56:59,615 INFO [Listener at localhost/40243] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-21 09:56:59,615 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,615 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,615 INFO [Listener at localhost/40243] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-21 09:56:59,615 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-21 09:56:59,615 INFO [Listener at localhost/40243] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-21 09:56:59,616 INFO [Listener at localhost/40243] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-21 09:56:59,617 INFO [Listener at localhost/40243] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41011 2023-02-21 09:56:59,617 INFO [Listener at localhost/40243] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-21 09:56:59,618 INFO [Listener at localhost/40243] mob.MobFileCache(124): MobFileCache disabled 2023-02-21 09:56:59,619 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,620 INFO [Listener at localhost/40243] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,621 INFO [Listener at localhost/40243] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41011 connecting to ZooKeeper ensemble=127.0.0.1:58856 2023-02-21 09:56:59,625 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:410110x0, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-21 09:56:59,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41011-0x100764f8e080003 connected 2023-02-21 09:56:59,626 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-21 09:56:59,627 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:56:59,627 DEBUG [Listener at localhost/40243] zookeeper.ZKUtil(164): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-21 09:56:59,629 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41011 2023-02-21 09:56:59,629 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41011 2023-02-21 09:56:59,630 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41011 2023-02-21 09:56:59,630 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41011 2023-02-21 09:56:59,631 DEBUG [Listener at localhost/40243] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41011 2023-02-21 09:56:59,632 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:56:59,641 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-21 09:56:59,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:56:59,660 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-21 09:56:59,660 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-21 09:56:59,660 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-21 09:56:59,660 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-21 09:56:59,660 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:56:59,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-21 09:56:59,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,41687,1676973418380 from backup master directory 2023-02-21 09:56:59,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-21 09:56:59,666 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:56:59,666 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-21 09:56:59,667 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-21 09:56:59,667 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:56:59,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-21 09:56:59,671 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-21 09:56:59,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/hbase.id with ID: 8cfec63e-3990-46fd-b6cf-640178c59293 2023-02-21 09:56:59,802 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-21 09:56:59,818 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:56:59,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0f853f68 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:56:59,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b927ddf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:56:59,936 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-21 09:56:59,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-21 09:56:59,962 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-21 09:56:59,962 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-21 09:56:59,965 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-21 09:56:59,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-21 09:56:59,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-21 09:57:00,005 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store-tmp 2023-02-21 09:57:00,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:00,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-21 09:57:00,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:00,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:00,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-21 09:57:00,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:00,041 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:00,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-21 09:57:00,043 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/WALs/jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:57:00,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41687%2C1676973418380, suffix=, logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/WALs/jenkins-hbase4.apache.org,41687,1676973418380, archiveDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/oldWALs, maxLogs=10 2023-02-21 09:57:00,116 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK] 2023-02-21 09:57:00,116 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK] 2023-02-21 09:57:00,116 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK] 2023-02-21 09:57:00,123 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-21 09:57:00,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/WALs/jenkins-hbase4.apache.org,41687,1676973418380/jenkins-hbase4.apache.org%2C41687%2C1676973418380.1676973420072 2023-02-21 09:57:00,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK], DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK], DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK]] 2023-02-21 09:57:00,197 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-21 09:57:00,197 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:00,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-21 09:57:00,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-21 09:57:00,259 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-21 09:57:00,267 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-21 09:57:00,299 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-21 09:57:00,310 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:00,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-21 09:57:00,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-21 09:57:00,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-21 09:57:00,335 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:00,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74655294, jitterRate=0.11245056986808777}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:00,336 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-21 09:57:00,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-21 09:57:00,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-21 09:57:00,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-21 09:57:00,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-21 09:57:00,361 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-21 09:57:00,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-21 09:57:00,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-21 09:57:00,417 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-21 09:57:00,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-21 09:57:00,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-21 09:57:00,448 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-21 09:57:00,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-21 09:57:00,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-21 09:57:00,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-21 09:57:00,461 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:00,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-21 09:57:00,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-21 09:57:00,475 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-21 09:57:00,479 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:00,480 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:00,479 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:00,479 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:00,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,41687,1676973418380, sessionid=0x100764f8e080000, setting cluster-up flag (Was=false) 2023-02-21 09:57:00,480 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:00,499 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:00,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-21 09:57:00,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:57:00,512 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:00,518 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-21 09:57:00,519 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:57:00,521 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp 2023-02-21 09:57:00,534 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(952): ClusterId : 8cfec63e-3990-46fd-b6cf-640178c59293 2023-02-21 09:57:00,535 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(952): ClusterId : 8cfec63e-3990-46fd-b6cf-640178c59293 2023-02-21 09:57:00,534 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(952): ClusterId : 8cfec63e-3990-46fd-b6cf-640178c59293 2023-02-21 09:57:00,540 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-21 09:57:00,540 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-21 09:57:00,540 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-21 09:57:00,547 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-21 09:57:00,547 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-21 09:57:00,547 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-21 09:57:00,547 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-21 09:57:00,547 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-21 09:57:00,547 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-21 09:57:00,552 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-21 09:57:00,553 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-21 09:57:00,553 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-21 09:57:00,554 DEBUG [RS:1;jenkins-hbase4:44835] zookeeper.ReadOnlyZKClient(139): Connect 0x2556f54f to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:00,555 DEBUG [RS:2;jenkins-hbase4:41011] zookeeper.ReadOnlyZKClient(139): Connect 0x2e195c33 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:00,555 DEBUG [RS:0;jenkins-hbase4:46077] zookeeper.ReadOnlyZKClient(139): Connect 0x5cf5f6aa to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:00,586 DEBUG [RS:1;jenkins-hbase4:44835] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b911f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:00,587 DEBUG [RS:2;jenkins-hbase4:41011] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20deb780, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:00,587 DEBUG [RS:2;jenkins-hbase4:41011] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24e13743, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-21 09:57:00,587 DEBUG [RS:1;jenkins-hbase4:44835] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fae9e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-21 09:57:00,589 DEBUG [RS:0;jenkins-hbase4:46077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52320344, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:00,589 DEBUG [RS:0;jenkins-hbase4:46077] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@560d9dda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-21 09:57:00,619 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:41011 2023-02-21 09:57:00,619 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:44835 2023-02-21 09:57:00,620 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46077 2023-02-21 09:57:00,623 INFO [RS:0;jenkins-hbase4:46077] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-21 09:57:00,623 INFO [RS:1;jenkins-hbase4:44835] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-21 09:57:00,624 INFO [RS:1;jenkins-hbase4:44835] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-21 09:57:00,623 INFO [RS:2;jenkins-hbase4:41011] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-21 09:57:00,624 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1023): About to register with Master. 2023-02-21 09:57:00,624 INFO [RS:0;jenkins-hbase4:46077] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-21 09:57:00,624 INFO [RS:2;jenkins-hbase4:41011] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-21 09:57:00,624 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1023): About to register with Master. 2023-02-21 09:57:00,624 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1023): About to register with Master. 2023-02-21 09:57:00,627 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41687,1676973418380 with isa=jenkins-hbase4.apache.org/172.31.14.131:41011, startcode=1676973419614 2023-02-21 09:57:00,627 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41687,1676973418380 with isa=jenkins-hbase4.apache.org/172.31.14.131:44835, startcode=1676973419584 2023-02-21 09:57:00,627 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41687,1676973418380 with isa=jenkins-hbase4.apache.org/172.31.14.131:46077, startcode=1676973419539 2023-02-21 09:57:00,645 DEBUG [RS:2;jenkins-hbase4:41011] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-21 09:57:00,645 DEBUG [RS:1;jenkins-hbase4:44835] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-21 09:57:00,645 DEBUG [RS:0;jenkins-hbase4:46077] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-21 09:57:00,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-21 09:57:00,703 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-21 09:57:00,703 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-21 09:57:00,703 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-21 09:57:00,704 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-21 09:57:00,704 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-21 09:57:00,704 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,704 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-21 09:57:00,704 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,704 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55535, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-21 09:57:00,704 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33555, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-21 09:57:00,704 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44837, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-21 09:57:00,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676973450712 2023-02-21 09:57:00,715 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-21 09:57:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41687] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41687] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:00,720 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-21 09:57:00,721 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-21 09:57:00,726 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-21 09:57:00,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-21 09:57:00,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-21 09:57:00,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-21 09:57:00,741 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-21 09:57:00,741 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-21 09:57:00,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,744 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-21 09:57:00,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-21 09:57:00,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-21 09:57:00,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-21 09:57:00,750 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-21 09:57:00,752 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676973420752,5,FailOnTimeoutGroup] 2023-02-21 09:57:00,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676973420752,5,FailOnTimeoutGroup] 2023-02-21 09:57:00,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,753 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-21 09:57:00,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,756 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(2829): Master is not running yet 2023-02-21 09:57:00,756 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(2829): Master is not running yet 2023-02-21 09:57:00,756 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(2829): Master is not running yet 2023-02-21 09:57:00,757 WARN [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-21 09:57:00,757 WARN [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-21 09:57:00,757 WARN [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-21 09:57:00,775 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:00,776 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:00,776 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926 2023-02-21 09:57:00,798 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:00,800 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-21 09:57:00,802 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/info 2023-02-21 09:57:00,803 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-21 09:57:00,804 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:00,804 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-21 09:57:00,806 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/rep_barrier 2023-02-21 09:57:00,807 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-21 09:57:00,807 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:00,807 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-21 09:57:00,809 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/table 2023-02-21 09:57:00,810 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-21 09:57:00,811 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:00,812 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740 2023-02-21 09:57:00,813 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740 2023-02-21 09:57:00,816 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-21 09:57:00,818 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-21 09:57:00,821 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:00,822 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59939264, jitterRate=-0.10683536529541016}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-21 09:57:00,822 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-21 09:57:00,822 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-21 09:57:00,822 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-21 09:57:00,822 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-21 09:57:00,822 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-21 09:57:00,822 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-21 09:57:00,823 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-21 09:57:00,824 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-21 09:57:00,828 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-21 09:57:00,828 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-21 09:57:00,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-21 09:57:00,849 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-21 09:57:00,852 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:00,858 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41687,1676973418380 with isa=jenkins-hbase4.apache.org/172.31.14.131:41011, startcode=1676973419614 2023-02-21 09:57:00,858 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41687,1676973418380 with isa=jenkins-hbase4.apache.org/172.31.14.131:44835, startcode=1676973419584 2023-02-21 09:57:00,858 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41687,1676973418380 with isa=jenkins-hbase4.apache.org/172.31.14.131:46077, startcode=1676973419539 2023-02-21 09:57:00,863 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41687] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:00,864 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41687] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:00,870 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926 2023-02-21 09:57:00,870 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926 2023-02-21 09:57:00,870 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926 2023-02-21 09:57:00,870 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35503 2023-02-21 09:57:00,870 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35503 2023-02-21 09:57:00,871 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-21 09:57:00,871 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35503 2023-02-21 09:57:00,871 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-21 09:57:00,871 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-21 09:57:00,879 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:00,880 DEBUG [RS:2;jenkins-hbase4:41011] zookeeper.ZKUtil(162): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,880 DEBUG [RS:1;jenkins-hbase4:44835] zookeeper.ZKUtil(162): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:00,880 WARN [RS:2;jenkins-hbase4:41011] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-21 09:57:00,880 WARN [RS:1;jenkins-hbase4:44835] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-21 09:57:00,881 INFO [RS:2;jenkins-hbase4:41011] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-21 09:57:00,881 DEBUG [RS:0;jenkins-hbase4:46077] zookeeper.ZKUtil(162): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:00,881 INFO [RS:1;jenkins-hbase4:44835] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-21 09:57:00,882 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,882 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:00,881 WARN [RS:0;jenkins-hbase4:46077] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-21 09:57:00,882 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41011,1676973419614] 2023-02-21 09:57:00,882 INFO [RS:0;jenkins-hbase4:46077] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-21 09:57:00,882 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44835,1676973419584] 2023-02-21 09:57:00,882 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:00,882 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46077,1676973419539] 2023-02-21 09:57:00,895 DEBUG [RS:0;jenkins-hbase4:46077] zookeeper.ZKUtil(162): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:00,895 DEBUG [RS:1;jenkins-hbase4:44835] zookeeper.ZKUtil(162): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:00,895 DEBUG [RS:2;jenkins-hbase4:41011] zookeeper.ZKUtil(162): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:00,896 DEBUG [RS:0;jenkins-hbase4:46077] zookeeper.ZKUtil(162): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,896 DEBUG [RS:1;jenkins-hbase4:44835] zookeeper.ZKUtil(162): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,896 DEBUG [RS:2;jenkins-hbase4:41011] zookeeper.ZKUtil(162): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,896 DEBUG [RS:0;jenkins-hbase4:46077] zookeeper.ZKUtil(162): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:00,896 DEBUG [RS:1;jenkins-hbase4:44835] zookeeper.ZKUtil(162): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:00,897 DEBUG [RS:2;jenkins-hbase4:41011] zookeeper.ZKUtil(162): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:00,906 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-21 09:57:00,906 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-21 09:57:00,906 DEBUG [RS:1;jenkins-hbase4:44835] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-21 09:57:00,915 INFO [RS:2;jenkins-hbase4:41011] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-21 09:57:00,915 INFO [RS:1;jenkins-hbase4:44835] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-21 09:57:00,915 INFO [RS:0;jenkins-hbase4:46077] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-21 09:57:00,935 INFO [RS:1;jenkins-hbase4:44835] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-21 09:57:00,935 INFO [RS:2;jenkins-hbase4:41011] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-21 09:57:00,935 INFO [RS:0;jenkins-hbase4:46077] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-21 09:57:00,939 INFO [RS:1;jenkins-hbase4:44835] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-21 09:57:00,939 INFO [RS:0;jenkins-hbase4:46077] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-21 09:57:00,939 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,939 INFO [RS:2;jenkins-hbase4:41011] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-21 09:57:00,940 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,940 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,940 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-21 09:57:00,940 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-21 09:57:00,940 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-21 09:57:00,948 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,948 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,948 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,948 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,948 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,950 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,950 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,950 DEBUG [RS:1;jenkins-hbase4:44835] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,950 DEBUG [RS:0;jenkins-hbase4:46077] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,949 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,952 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,952 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,952 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,952 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,953 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-21 09:57:00,953 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,953 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,953 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,953 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,953 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,953 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,953 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,953 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,953 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,954 DEBUG [RS:2;jenkins-hbase4:41011] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-21 09:57:00,959 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,959 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,959 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,970 INFO [RS:0;jenkins-hbase4:46077] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-21 09:57:00,970 INFO [RS:1;jenkins-hbase4:44835] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-21 09:57:00,973 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46077,1676973419539-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,973 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44835,1676973419584-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,973 INFO [RS:2;jenkins-hbase4:41011] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-21 09:57:00,973 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41011,1676973419614-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:00,996 INFO [RS:0;jenkins-hbase4:46077] regionserver.Replication(203): jenkins-hbase4.apache.org,46077,1676973419539 started 2023-02-21 09:57:00,996 INFO [RS:2;jenkins-hbase4:41011] regionserver.Replication(203): jenkins-hbase4.apache.org,41011,1676973419614 started 2023-02-21 09:57:00,996 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46077,1676973419539, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46077, sessionid=0x100764f8e080001 2023-02-21 09:57:00,996 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41011,1676973419614, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41011, sessionid=0x100764f8e080003 2023-02-21 09:57:00,999 INFO [RS:1;jenkins-hbase4:44835] regionserver.Replication(203): jenkins-hbase4.apache.org,44835,1676973419584 started 2023-02-21 09:57:00,999 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-21 09:57:00,999 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-21 09:57:00,999 DEBUG [RS:2;jenkins-hbase4:41011] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:00,999 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44835,1676973419584, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44835, sessionid=0x100764f8e080002 2023-02-21 09:57:01,000 DEBUG [RS:2;jenkins-hbase4:41011] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41011,1676973419614' 2023-02-21 09:57:00,999 DEBUG [RS:0;jenkins-hbase4:46077] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:01,000 DEBUG [RS:2;jenkins-hbase4:41011] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-21 09:57:01,000 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-21 09:57:01,000 DEBUG [RS:0;jenkins-hbase4:46077] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46077,1676973419539' 2023-02-21 09:57:01,001 DEBUG [RS:0;jenkins-hbase4:46077] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-21 09:57:01,001 DEBUG [RS:1;jenkins-hbase4:44835] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:01,001 DEBUG [RS:1;jenkins-hbase4:44835] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44835,1676973419584' 2023-02-21 09:57:01,001 DEBUG [RS:1;jenkins-hbase4:44835] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-21 09:57:01,001 DEBUG [RS:2;jenkins-hbase4:41011] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-21 09:57:01,001 DEBUG [RS:0;jenkins-hbase4:46077] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-21 09:57:01,001 DEBUG [RS:1;jenkins-hbase4:44835] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-21 09:57:01,002 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-21 09:57:01,002 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-21 09:57:01,002 DEBUG [RS:2;jenkins-hbase4:41011] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:01,002 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-21 09:57:01,002 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-21 09:57:01,002 DEBUG [RS:2;jenkins-hbase4:41011] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41011,1676973419614' 2023-02-21 09:57:01,002 DEBUG [RS:2;jenkins-hbase4:41011] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-21 09:57:01,002 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-21 09:57:01,002 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-21 09:57:01,003 DEBUG [RS:0;jenkins-hbase4:46077] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:01,003 DEBUG [RS:0;jenkins-hbase4:46077] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46077,1676973419539' 2023-02-21 09:57:01,003 DEBUG [RS:0;jenkins-hbase4:46077] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-21 09:57:01,002 DEBUG [RS:1;jenkins-hbase4:44835] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:01,003 DEBUG [RS:1;jenkins-hbase4:44835] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44835,1676973419584' 2023-02-21 09:57:01,003 DEBUG [RS:1;jenkins-hbase4:44835] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-21 09:57:01,003 DEBUG [RS:1;jenkins-hbase4:44835] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-21 09:57:01,004 DEBUG [RS:1;jenkins-hbase4:44835] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-21 09:57:01,004 INFO [RS:1;jenkins-hbase4:44835] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-21 09:57:01,004 INFO [RS:1;jenkins-hbase4:44835] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-21 09:57:01,005 DEBUG [RS:2;jenkins-hbase4:41011] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-21 09:57:01,005 DEBUG [RS:0;jenkins-hbase4:46077] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-21 09:57:01,005 DEBUG [jenkins-hbase4:41687] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-21 09:57:01,005 DEBUG [RS:2;jenkins-hbase4:41011] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-21 09:57:01,005 DEBUG [RS:0;jenkins-hbase4:46077] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-21 09:57:01,005 INFO [RS:2;jenkins-hbase4:41011] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-21 09:57:01,005 INFO [RS:0;jenkins-hbase4:46077] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-21 09:57:01,009 INFO [RS:2;jenkins-hbase4:41011] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-21 09:57:01,009 INFO [RS:0;jenkins-hbase4:46077] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-21 09:57:01,010 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:01,015 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:01,015 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:01,015 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:01,015 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:01,018 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46077,1676973419539, state=OPENING 2023-02-21 09:57:01,024 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-21 09:57:01,026 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:01,026 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-21 09:57:01,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:01,115 INFO [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46077%2C1676973419539, suffix=, logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,46077,1676973419539, archiveDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs, maxLogs=32 2023-02-21 09:57:01,115 INFO [RS:1;jenkins-hbase4:44835] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44835%2C1676973419584, suffix=, logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,44835,1676973419584, archiveDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs, maxLogs=32 2023-02-21 09:57:01,115 INFO [RS:2;jenkins-hbase4:41011] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41011%2C1676973419614, suffix=, logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,41011,1676973419614, archiveDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs, maxLogs=32 2023-02-21 09:57:01,136 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK] 2023-02-21 09:57:01,136 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK] 2023-02-21 09:57:01,137 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK] 2023-02-21 09:57:01,147 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK] 2023-02-21 09:57:01,147 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK] 2023-02-21 09:57:01,148 INFO [RS:1;jenkins-hbase4:44835] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,44835,1676973419584/jenkins-hbase4.apache.org%2C44835%2C1676973419584.1676973421119 2023-02-21 09:57:01,150 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK] 2023-02-21 09:57:01,153 DEBUG [RS:1;jenkins-hbase4:44835] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK], DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK], DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK]] 2023-02-21 09:57:01,158 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK] 2023-02-21 09:57:01,158 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK] 2023-02-21 09:57:01,158 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK] 2023-02-21 09:57:01,163 INFO [RS:2;jenkins-hbase4:41011] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,41011,1676973419614/jenkins-hbase4.apache.org%2C41011%2C1676973419614.1676973421119 2023-02-21 09:57:01,166 DEBUG [RS:2;jenkins-hbase4:41011] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK], DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK], DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK]] 2023-02-21 09:57:01,166 INFO [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,46077,1676973419539/jenkins-hbase4.apache.org%2C46077%2C1676973419539.1676973421119 2023-02-21 09:57:01,168 DEBUG [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK], DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK], DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK]] 2023-02-21 09:57:01,213 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:01,216 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-21 09:57:01,219 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46632, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-21 09:57:01,231 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-21 09:57:01,231 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-21 09:57:01,235 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46077%2C1676973419539.meta, suffix=.meta, logDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,46077,1676973419539, archiveDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs, maxLogs=32 2023-02-21 09:57:01,257 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK] 2023-02-21 09:57:01,257 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK] 2023-02-21 09:57:01,260 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK] 2023-02-21 09:57:01,266 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/WALs/jenkins-hbase4.apache.org,46077,1676973419539/jenkins-hbase4.apache.org%2C46077%2C1676973419539.meta.1676973421237.meta 2023-02-21 09:57:01,266 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44327,DS-8931b049-6053-42e9-9bd0-e03684384fda,DISK], DatanodeInfoWithStorage[127.0.0.1:33051,DS-052ab17e-09a8-4f09-8081-ac55350de361,DISK], DatanodeInfoWithStorage[127.0.0.1:39513,DS-3c068c46-ef90-4903-925d-b2597b1e20eb,DISK]] 2023-02-21 09:57:01,267 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-21 09:57:01,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-21 09:57:01,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-21 09:57:01,292 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-21 09:57:01,297 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-21 09:57:01,297 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:01,297 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-21 09:57:01,297 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-21 09:57:01,300 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-21 09:57:01,301 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/info 2023-02-21 09:57:01,302 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/info 2023-02-21 09:57:01,302 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-21 09:57:01,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:01,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-21 09:57:01,305 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/rep_barrier 2023-02-21 09:57:01,305 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/rep_barrier 2023-02-21 09:57:01,305 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-21 09:57:01,306 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:01,306 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-21 09:57:01,307 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/table 2023-02-21 09:57:01,307 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/table 2023-02-21 09:57:01,308 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-21 09:57:01,309 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:01,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740 2023-02-21 09:57:01,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740 2023-02-21 09:57:01,317 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-21 09:57:01,319 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-21 09:57:01,320 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75225002, jitterRate=0.12093988060951233}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-21 09:57:01,321 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-21 09:57:01,330 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676973421206 2023-02-21 09:57:01,347 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-21 09:57:01,348 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-21 09:57:01,348 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46077,1676973419539, state=OPEN 2023-02-21 09:57:01,352 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-21 09:57:01,352 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-21 09:57:01,356 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-21 09:57:01,356 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46077,1676973419539 in 323 msec 2023-02-21 09:57:01,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-21 09:57:01,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 522 msec 2023-02-21 09:57:01,368 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 758 msec 2023-02-21 09:57:01,368 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1676973421368, completionTime=-1 2023-02-21 09:57:01,368 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-21 09:57:01,368 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-21 09:57:01,433 DEBUG [hconnection-0x72233ac4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:01,436 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:01,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-21 09:57:01,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676973481452 2023-02-21 09:57:01,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676973541453 2023-02-21 09:57:01,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 84 msec 2023-02-21 09:57:01,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41687,1676973418380-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:01,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41687,1676973418380-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:01,479 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41687,1676973418380-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:01,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:41687, period=300000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:01,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-21 09:57:01,491 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-21 09:57:01,500 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-21 09:57:01,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-21 09:57:01,512 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-21 09:57:01,514 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-21 09:57:01,517 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-21 09:57:01,540 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:01,542 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688 empty. 2023-02-21 09:57:01,543 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:01,543 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-21 09:57:01,581 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:01,583 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 46e4fb13fce46594ac37583388eb6688, NAME => 'hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:01,603 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:01,604 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 46e4fb13fce46594ac37583388eb6688, disabling compactions & flushes 2023-02-21 09:57:01,604 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:01,604 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:01,604 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. after waiting 0 ms 2023-02-21 09:57:01,604 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:01,604 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:01,604 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 46e4fb13fce46594ac37583388eb6688: 2023-02-21 09:57:01,609 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-21 09:57:01,624 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676973421612"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973421612"}]},"ts":"1676973421612"} 2023-02-21 09:57:01,650 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-21 09:57:01,652 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-21 09:57:01,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973421652"}]},"ts":"1676973421652"} 2023-02-21 09:57:01,661 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-21 09:57:01,667 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:01,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:01,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:01,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:01,668 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:01,671 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=46e4fb13fce46594ac37583388eb6688, ASSIGN}] 2023-02-21 09:57:01,674 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=46e4fb13fce46594ac37583388eb6688, ASSIGN 2023-02-21 09:57:01,675 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=46e4fb13fce46594ac37583388eb6688, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:01,828 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-21 09:57:01,828 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=46e4fb13fce46594ac37583388eb6688, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:01,829 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676973421828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973421828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973421828"}]},"ts":"1676973421828"} 2023-02-21 09:57:01,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 46e4fb13fce46594ac37583388eb6688, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:01,992 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:01,992 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-21 09:57:01,997 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34468, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-21 09:57:02,003 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:02,005 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 46e4fb13fce46594ac37583388eb6688, NAME => 'hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.', STARTKEY => '', ENDKEY => ''} 2023-02-21 09:57:02,005 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,005 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,006 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,006 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,007 INFO [StoreOpener-46e4fb13fce46594ac37583388eb6688-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,010 DEBUG [StoreOpener-46e4fb13fce46594ac37583388eb6688-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/info 2023-02-21 09:57:02,010 DEBUG [StoreOpener-46e4fb13fce46594ac37583388eb6688-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/info 2023-02-21 09:57:02,010 INFO [StoreOpener-46e4fb13fce46594ac37583388eb6688-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 46e4fb13fce46594ac37583388eb6688 columnFamilyName info 2023-02-21 09:57:02,011 INFO [StoreOpener-46e4fb13fce46594ac37583388eb6688-1] regionserver.HStore(310): Store=46e4fb13fce46594ac37583388eb6688/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,012 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,013 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,017 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:02,021 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,022 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 46e4fb13fce46594ac37583388eb6688; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64169174, jitterRate=-0.04380479454994202}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,022 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 46e4fb13fce46594ac37583388eb6688: 2023-02-21 09:57:02,025 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688., pid=6, masterSystemTime=1676973421992 2023-02-21 09:57:02,029 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:02,030 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:02,031 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=46e4fb13fce46594ac37583388eb6688, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:02,031 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676973422030"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422030"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422030"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422030"}]},"ts":"1676973422030"} 2023-02-21 09:57:02,040 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-21 09:57:02,040 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 46e4fb13fce46594ac37583388eb6688, server=jenkins-hbase4.apache.org,41011,1676973419614 in 199 msec 2023-02-21 09:57:02,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-21 09:57:02,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=46e4fb13fce46594ac37583388eb6688, ASSIGN in 369 msec 2023-02-21 09:57:02,049 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-21 09:57:02,050 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973422050"}]},"ts":"1676973422050"} 2023-02-21 09:57:02,052 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-21 09:57:02,055 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-21 09:57:02,058 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 553 msec 2023-02-21 09:57:02,115 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-21 09:57:02,116 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-21 09:57:02,116 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:02,137 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:02,138 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34470, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:02,151 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-21 09:57:02,167 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-21 09:57:02,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-02-21 09:57:02,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-21 09:57:02,195 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-21 09:57:02,200 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-02-21 09:57:02,210 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-21 09:57:02,213 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-21 09:57:02,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.546sec 2023-02-21 09:57:02,216 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-21 09:57:02,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-21 09:57:02,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-21 09:57:02,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41687,1676973418380-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-21 09:57:02,220 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41687,1676973418380-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-21 09:57:02,228 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-02-21 09:57:02,244 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(139): Connect 0x1c633018 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:02,248 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b210801, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:02,263 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:02,305 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46650, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:02,315 INFO [Listener at localhost/40243] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:57:02,345 INFO [Listener at localhost/40243] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=360, OpenFileDescriptor=697, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=174, AvailableMemoryMB=5652 2023-02-21 09:57:02,360 DEBUG [Listener at localhost/40243] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-21 09:57:02,363 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43920, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-21 09:57:02,375 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-21 09:57:02,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:02,380 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676973422348 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-21 09:57:02,382 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676973422348 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-21 09:57:02,384 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676973422348" procId is: 9 2023-02-21 09:57:02,390 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,391 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,391 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,391 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,391 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,391 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 empty. 2023-02-21 09:57:02,391 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 empty. 2023-02-21 09:57:02,394 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc empty. 2023-02-21 09:57:02,394 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 empty. 2023-02-21 09:57:02,394 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-21 09:57:02,392 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,395 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,395 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 empty. 2023-02-21 09:57:02,395 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,396 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,396 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,395 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 empty. 2023-02-21 09:57:02,398 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,400 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676973422348 regions 2023-02-21 09:57:02,423 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:02,425 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(7671): creating {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:02,425 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(7671): creating {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:02,425 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(7671): creating {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:02,475 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,475 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:02,476 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,476 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:02,476 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,477 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,476 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,477 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,477 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:02,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:02,478 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(7671): creating {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:02,478 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(7671): creating {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:02,478 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(7671): creating {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:02,504 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:02,505 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:02,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,505 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-2] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:02,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,506 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:02,506 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,506 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,506 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:02,506 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,506 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,506 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-1] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:02,506 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,507 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:02,507 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,507 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,507 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:02,507 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,507 INFO [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,507 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676973422348-pool-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:02,512 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676973422348 execute state=CREATE_TABLE_ADD_TO_META 2023-02-21 09:57:02,513 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973422513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973422513"}]},"ts":"1676973422513"} 2023-02-21 09:57:02,514 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973422513"}]},"ts":"1676973422513"} 2023-02-21 09:57:02,514 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973422513"}]},"ts":"1676973422513"} 2023-02-21 09:57:02,514 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973422513"}]},"ts":"1676973422513"} 2023-02-21 09:57:02,514 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973422513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973422513"}]},"ts":"1676973422513"} 2023-02-21 09:57:02,514 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973422513"}]},"ts":"1676973422513"} 2023-02-21 09:57:02,557 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-21 09:57:02,559 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676973422348 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-21 09:57:02,559 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973422559"}]},"ts":"1676973422559"} 2023-02-21 09:57:02,561 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:02,567 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:02,572 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:02,572 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:02,572 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:02,572 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:02,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:02,576 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:02,576 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:02,577 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:02,577 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:02,578 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=false 2023-02-21 09:57:02,578 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:02,578 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=false 2023-02-21 09:57:02,578 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:02,578 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=false 2023-02-21 09:57:02,579 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:02,580 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=false 2023-02-21 09:57:02,580 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-21 09:57:02,728 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:02,731 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:02,731 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:02,731 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:02,731 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:02,731 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:02,731 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973422731"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973422731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973422731"}]},"ts":"1676973422731"} 2023-02-21 09:57:02,731 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422731"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973422731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973422731"}]},"ts":"1676973422731"} 2023-02-21 09:57:02,731 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422731"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973422731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973422731"}]},"ts":"1676973422731"} 2023-02-21 09:57:02,731 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422731"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973422731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973422731"}]},"ts":"1676973422731"} 2023-02-21 09:57:02,731 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422731"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973422731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973422731"}]},"ts":"1676973422731"} 2023-02-21 09:57:02,734 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:02,736 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:02,738 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=13, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:02,739 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=14, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:02,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=11, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:02,748 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:02,748 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973422748"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973422748"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973422748"}]},"ts":"1676973422748"} 2023-02-21 09:57:02,751 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=10, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:02,888 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:02,888 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-21 09:57:02,891 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39090, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-21 09:57:02,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:02,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,901 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,904 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,904 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:02,906 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,906 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,908 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:02,908 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:02,909 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:02,909 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:02,910 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:02,911 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:02,911 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:02,913 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,916 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,913 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:02,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:02,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:02,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60178867, jitterRate=-0.10326500236988068}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:02,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64824966, jitterRate=-0.034032732248306274}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:02,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60773485, jitterRate=-0.09440450370311737}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:02,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=18, masterSystemTime=1676973422898 2023-02-21 09:57:02,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=19, masterSystemTime=1676973422898 2023-02-21 09:57:02,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=17, masterSystemTime=1676973422888 2023-02-21 09:57:02,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:02,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:02,945 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:02,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,945 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422945"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422945"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422945"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422945"}]},"ts":"1676973422945"} 2023-02-21 09:57:02,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:02,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,947 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:02,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:02,947 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422947"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422947"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422947"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422947"}]},"ts":"1676973422947"} 2023-02-21 09:57:02,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:02,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:02,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:02,951 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,951 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:02,952 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422951"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422951"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422951"}]},"ts":"1676973422951"} 2023-02-21 09:57:02,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,954 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,957 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:02,957 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:02,957 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:02,957 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:02,958 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:02,958 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:02,959 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=13 2023-02-21 09:57:02,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=13, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 212 msec 2023-02-21 09:57:02,961 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,961 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,962 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=14 2023-02-21 09:57:02,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=14, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 215 msec 2023-02-21 09:57:02,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,965 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 386 msec 2023-02-21 09:57:02,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,966 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-02-21 09:57:02,966 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 223 msec 2023-02-21 09:57:02,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 390 msec 2023-02-21 09:57:02,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 394 msec 2023-02-21 09:57:02,969 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:02,969 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:02,970 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:02,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:02,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:02,972 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75418903, jitterRate=0.12382923066616058}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:02,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=20, masterSystemTime=1676973422898 2023-02-21 09:57:02,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73585432, jitterRate=0.09650838375091553}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:02,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=16, masterSystemTime=1676973422888 2023-02-21 09:57:02,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:02,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:02,982 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:02,982 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973422982"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422982"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422982"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422982"}]},"ts":"1676973422982"} 2023-02-21 09:57:02,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:02,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:02,987 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:02,987 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973422986"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422986"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422986"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422986"}]},"ts":"1676973422986"} 2023-02-21 09:57:02,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69821525, jitterRate=0.04042179882526398}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:02,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:02,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=21, masterSystemTime=1676973422898 2023-02-21 09:57:02,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=11 2023-02-21 09:57:02,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=11, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 246 msec 2023-02-21 09:57:02,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:02,993 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:02,994 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973422993"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973422993"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973422993"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973422993"}]},"ts":"1676973422993"} 2023-02-21 09:57:02,995 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-02-21 09:57:02,995 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 419 msec 2023-02-21 09:57:02,995 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 257 msec 2023-02-21 09:57:02,998 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 423 msec 2023-02-21 09:57:03,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=10 2023-02-21 09:57:03,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=10, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 246 msec 2023-02-21 09:57:03,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-21 09:57:03,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 427 msec 2023-02-21 09:57:03,004 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676973422348 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-21 09:57:03,005 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973423004"}]},"ts":"1676973423004"} 2023-02-21 09:57:03,006 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:03,010 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676973422348 execute state=CREATE_TABLE_POST_OPERATION 2023-02-21 09:57:03,012 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676973422348 in 634 msec 2023-02-21 09:57:03,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-21 09:57:03,151 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 9 completed 2023-02-21 09:57:03,157 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,158 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:03,161 INFO [Listener at localhost/40243] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-21 09:57:03,182 DEBUG [hconnection-0x7b9b1635-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:03,186 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34482, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:03,190 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:03,193 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39092, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:03,203 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,208 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,219 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973423219"}]},"ts":"1676973423219"} 2023-02-21 09:57:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-21 09:57:03,222 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:03,224 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:03,227 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:03,230 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:03,230 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:03,230 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:03,231 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:03,231 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:03,233 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:03,233 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:03,233 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:03,233 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:03,233 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973423232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423232"}]},"ts":"1676973423232"} 2023-02-21 09:57:03,233 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423233"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423233"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423233"}]},"ts":"1676973423233"} 2023-02-21 09:57:03,233 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423233"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423233"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423233"}]},"ts":"1676973423233"} 2023-02-21 09:57:03,233 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:03,233 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973423233"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423233"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423233"}]},"ts":"1676973423233"} 2023-02-21 09:57:03,233 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423233"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423233"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423233"}]},"ts":"1676973423233"} 2023-02-21 09:57:03,235 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:03,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:03,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=24, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:03,239 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=28, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:03,240 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=27, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:03,246 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:03,247 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:03,248 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423247"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423247"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423247"}]},"ts":"1676973423247"} 2023-02-21 09:57:03,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=26, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:03,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:03,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:03,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:03,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:03,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:03,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:03,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:03,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:03,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:03,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:03,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:03,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:03,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:03,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:03,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:03,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:03,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:03,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:03,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:03,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:03,413 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:03,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:03,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:03,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:03,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:03,414 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423413"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973423413"}]},"ts":"1676973423413"} 2023-02-21 09:57:03,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:03,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:03,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:03,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:03,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:03,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:03,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:03,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:03,418 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:03,418 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973423418"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973423418"}]},"ts":"1676973423418"} 2023-02-21 09:57:03,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:03,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:03,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:03,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:03,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:03,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:03,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:03,419 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:03,420 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423419"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973423419"}]},"ts":"1676973423419"} 2023-02-21 09:57:03,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:03,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-21 09:57:03,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 183 msec 2023-02-21 09:57:03,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:03,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:03,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:03,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:03,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:03,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:03,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:03,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:03,427 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-21 09:57:03,427 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 187 msec 2023-02-21 09:57:03,428 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 198 msec 2023-02-21 09:57:03,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:03,429 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=24 2023-02-21 09:57:03,429 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=24, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 186 msec 2023-02-21 09:57:03,429 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:03,429 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423429"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973423429"}]},"ts":"1676973423429"} 2023-02-21 09:57:03,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:03,430 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 200 msec 2023-02-21 09:57:03,430 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:03,431 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973423430"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973423430"}]},"ts":"1676973423430"} 2023-02-21 09:57:03,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:03,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 202 msec 2023-02-21 09:57:03,433 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:03,433 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973423433"}]},"ts":"1676973423433"} 2023-02-21 09:57:03,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=27 2023-02-21 09:57:03,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=27, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 191 msec 2023-02-21 09:57:03,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-02-21 09:57:03,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 194 msec 2023-02-21 09:57:03,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 208 msec 2023-02-21 09:57:03,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=26 2023-02-21 09:57:03,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=26, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 185 msec 2023-02-21 09:57:03,438 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 209 msec 2023-02-21 09:57:03,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=22 2023-02-21 09:57:03,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 211 msec 2023-02-21 09:57:03,442 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973423441"}]},"ts":"1676973423441"} 2023-02-21 09:57:03,443 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:03,446 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:03,449 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 238 msec 2023-02-21 09:57:03,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-21 09:57:03,472 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 22 completed 2023-02-21 09:57:03,483 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973423483 (current time:1676973423483). 2023-02-21 09:57:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x317e5042 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:03,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44c53a33, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:03,498 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36992, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x317e5042 to 127.0.0.1:58856 2023-02-21 09:57:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:03,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:03,523 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:03,524 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE; timeout=600000, timestamp=1676974023524 2023-02-21 09:57:03,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:03,524 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676973422348 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,530 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:03,530 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:03,547 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:03,553 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,553 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,553 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,556 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,556 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,557 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,557 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,560 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:03,567 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,568 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,568 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,569 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,570 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,571 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,572 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:03,669 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:03,670 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676973422348 2023-02-21 09:57:03,672 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973422348 2023-02-21 09:57:03,723 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973422348 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/emptySnaptb-1676973422348 2023-02-21 09:57:03,760 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676973422348 of table testRestoreSnapshot-1676973422348 completed 2023-02-21 09:57:03,760 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973422348 2023-02-21 09:57:03,761 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973422348 2023-02-21 09:57:03,761 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676973422348 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 at 1676973423524Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } at 1676973423548 (+24 ms)Consolidate snapshot: emptySnaptb-1676973422348 at 1676973423670 (+122 ms)Loading Region manifests for emptySnaptb-1676973422348 at 1676973423670Writing data manifest for emptySnaptb-1676973422348 at 1676973423689 (+19 ms)Verifying snapshot: emptySnaptb-1676973422348 at 1676973423709 (+20 ms)Snapshot emptySnaptb-1676973422348 of table testRestoreSnapshot-1676973422348 completed at 1676973423760 (+51 ms) 2023-02-21 09:57:03,762 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:03,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE in 248 msec 2023-02-21 09:57:03,780 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:03,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:03,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:03,787 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,791 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-21 09:57:03,802 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676973422348 2023-02-21 09:57:03,802 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973423802"}]},"ts":"1676973423802"} 2023-02-21 09:57:03,804 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:03,807 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:03,809 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:03,811 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:03,811 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:03,812 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:03,812 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:03,813 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:03,813 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:03,813 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:03,814 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:03,814 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:03,814 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:03,814 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:03,815 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:03,815 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:03,815 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:03,815 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:03,815 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:03,816 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:03,816 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:03,963 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:03,964 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:03,964 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:03,964 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:03,964 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:03,964 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423964"}]},"ts":"1676973423964"} 2023-02-21 09:57:03,964 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423964"}]},"ts":"1676973423964"} 2023-02-21 09:57:03,964 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:03,964 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423964"}]},"ts":"1676973423964"} 2023-02-21 09:57:03,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973423964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423964"}]},"ts":"1676973423964"} 2023-02-21 09:57:03,964 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973423964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423964"}]},"ts":"1676973423964"} 2023-02-21 09:57:03,967 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=38, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:03,968 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=40, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:03,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=41, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:03,970 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=39, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:03,973 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=42, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:03,973 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:03,974 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973423973"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973423973"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973423973"}]},"ts":"1676973423973"} 2023-02-21 09:57:03,976 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=37, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-21 09:57:04,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:04,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:04,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:04,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,127 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,128 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:04,128 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:04,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:04,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:04,129 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:04,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:04,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:04,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:04,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:04,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,131 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:04,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,132 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,132 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,133 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:04,134 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:04,134 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:04,134 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:04,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,135 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:04,135 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:04,136 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:04,137 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:04,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:04,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63413182, jitterRate=-0.055069953203201294}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:04,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:04,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=43, masterSystemTime=1676973424119 2023-02-21 09:57:04,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:04,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:04,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:04,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:04,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:04,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,149 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:04,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66025384, jitterRate=-0.01614511013031006}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:04,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:04,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72728696, jitterRate=0.08374202251434326}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:04,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:04,149 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424149"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973424149"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973424149"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973424149"}]},"ts":"1676973424149"} 2023-02-21 09:57:04,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=47, masterSystemTime=1676973424125 2023-02-21 09:57:04,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=45, masterSystemTime=1676973424125 2023-02-21 09:57:04,153 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:04,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:04,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:04,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:04,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,155 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:04,155 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:04,155 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:04,155 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424155"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973424155"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973424155"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973424155"}]},"ts":"1676973424155"} 2023-02-21 09:57:04,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:04,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,156 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:04,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:04,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:04,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:04,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:04,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:04,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,160 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:04,161 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973424160"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973424160"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973424160"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973424160"}]},"ts":"1676973424160"} 2023-02-21 09:57:04,161 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=38 2023-02-21 09:57:04,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=38, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 186 msec 2023-02-21 09:57:04,162 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,162 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:04,164 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,165 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:04,165 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:04,166 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:04,167 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 355 msec 2023-02-21 09:57:04,167 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:04,168 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:04,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,168 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:04,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=41 2023-02-21 09:57:04,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=41, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 193 msec 2023-02-21 09:57:04,169 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:04,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=42 2023-02-21 09:57:04,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=42, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 191 msec 2023-02-21 09:57:04,170 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:04,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,172 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 362 msec 2023-02-21 09:57:04,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 363 msec 2023-02-21 09:57:04,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66916646, jitterRate=-0.002864271402359009}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:04,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:04,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=44, masterSystemTime=1676973424119 2023-02-21 09:57:04,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73371308, jitterRate=0.09331768751144409}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:04,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:04,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:04,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=46, masterSystemTime=1676973424125 2023-02-21 09:57:04,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:04,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64778118, jitterRate=-0.03473082184791565}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:04,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:04,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:04,183 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:04,183 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424183"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973424183"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973424183"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973424183"}]},"ts":"1676973424183"} 2023-02-21 09:57:04,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=48, masterSystemTime=1676973424125 2023-02-21 09:57:04,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:04,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:04,189 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:04,189 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424188"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973424188"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973424188"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973424188"}]},"ts":"1676973424188"} 2023-02-21 09:57:04,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:04,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:04,190 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:04,191 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973424190"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973424190"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973424190"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973424190"}]},"ts":"1676973424190"} 2023-02-21 09:57:04,192 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=40 2023-02-21 09:57:04,192 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=40, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 218 msec 2023-02-21 09:57:04,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=39 2023-02-21 09:57:04,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=39, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 221 msec 2023-02-21 09:57:04,195 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 385 msec 2023-02-21 09:57:04,197 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=37 2023-02-21 09:57:04,197 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 388 msec 2023-02-21 09:57:04,197 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=37, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 217 msec 2023-02-21 09:57:04,199 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-02-21 09:57:04,199 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 390 msec 2023-02-21 09:57:04,200 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973424200"}]},"ts":"1676973424200"} 2023-02-21 09:57:04,202 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:04,205 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676973422348' was successfully enabled. 2023-02-21 09:57:04,208 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676973422348 in 414 msec 2023-02-21 09:57:04,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-21 09:57:04,554 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 36 completed 2023-02-21 09:57:04,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:04,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:04,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:04,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:04,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:04,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:04,624 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676973422348 2023-02-21 09:57:04,624 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:04,625 INFO [Listener at localhost/40243] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-21 09:57:04,645 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:04,647 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:04,650 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:04,652 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:04,654 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:04,665 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:04,665 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:04,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:04,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-21 09:57:04,672 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973424672"}]},"ts":"1676973424672"} 2023-02-21 09:57:04,673 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:04,675 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:04,676 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:04,678 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:04,679 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:04,679 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:04,679 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:04,679 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:04,680 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:04,680 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:04,680 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973424680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973424680"}]},"ts":"1676973424680"} 2023-02-21 09:57:04,680 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:04,680 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973424680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973424680"}]},"ts":"1676973424680"} 2023-02-21 09:57:04,680 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973424680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973424680"}]},"ts":"1676973424680"} 2023-02-21 09:57:04,680 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:04,680 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:04,680 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973424680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973424680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973424680"}]},"ts":"1676973424680"} 2023-02-21 09:57:04,680 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973424680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973424680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973424680"}]},"ts":"1676973424680"} 2023-02-21 09:57:04,682 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=51, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:04,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=52, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:04,684 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=54, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:04,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=55, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:04,687 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=53, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:04,688 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:04,689 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:04,689 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973424689"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973424689"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973424689"}]},"ts":"1676973424689"} 2023-02-21 09:57:04,691 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=50, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:04,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:04,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:04,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:04,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:04,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:04,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:04,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:04,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:04,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:04,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:04,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:04,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:04,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5c80767ec66fcdbcf22028d0c0cfeee5 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-21 09:57:04,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5863e48176a446e50692d5bcf8d380f1 1/1 column families, dataSize=23.55 KB heapSize=50.88 KB 2023-02-21 09:57:04,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 74b647ed77adba33d8bf8ed03d9c05fc 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-21 09:57:04,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-21 09:57:04,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:04,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:04,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:04,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/.tmp/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:04,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/.tmp/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:04,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/.tmp/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:05,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/.tmp/cf/d65bd76661554d98a17951ff943be720 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:05,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/.tmp/cf/2d39b975032e413ab7e72e6533951b3e as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:05,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/.tmp/cf/1d0a8ea5761f44caa7dda024184b0758 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:05,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720, entries=38, sequenceid=8, filesize=11.4 K 2023-02-21 09:57:05,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e, entries=25, sequenceid=8, filesize=9.1 K 2023-02-21 09:57:05,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 5c80767ec66fcdbcf22028d0c0cfeee5 in 176ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:05,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 74b647ed77adba33d8bf8ed03d9c05fc in 176ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:05,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758, entries=360, sequenceid=8, filesize=66.2 K 2023-02-21 09:57:05,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676973422348' 2023-02-21 09:57:05,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676973422348' 2023-02-21 09:57:05,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.55 KB/24116, heapSize ~50.86 KB/52080, currentSize=0 B/0 for 5863e48176a446e50692d5bcf8d380f1 in 180ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:05,018 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676973422348' 2023-02-21 09:57:05,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:05,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:05,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:05,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:05,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:05,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:05,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:05,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:05,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:05,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:05,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:05,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:05,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:05,033 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:05,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a585cd7f02d60b76c310aa80c70f3a08 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-21 09:57:05,033 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973425033"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973425033"}]},"ts":"1676973425033"} 2023-02-21 09:57:05,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:05,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:05,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:05,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:05,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:05,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:05,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:05,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing dbcdd07ed31c76cf5ed03a249f4cf8d1 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-21 09:57:05,037 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:05,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:05,037 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425037"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973425037"}]},"ts":"1676973425037"} 2023-02-21 09:57:05,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:05,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:05,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:05,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:05,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:05,039 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:05,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:05,039 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973425039"}]},"ts":"1676973425039"} 2023-02-21 09:57:05,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing eb310106c5f65634673574312633a7b9 1/1 column families, dataSize=1.37 KB heapSize=3.20 KB 2023-02-21 09:57:05,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=55 2023-02-21 09:57:05,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=55, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 350 msec 2023-02-21 09:57:05,046 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 367 msec 2023-02-21 09:57:05,047 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=54 2023-02-21 09:57:05,047 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=54, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 357 msec 2023-02-21 09:57:05,047 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=53 2023-02-21 09:57:05,047 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=53, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 356 msec 2023-02-21 09:57:05,054 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 371 msec 2023-02-21 09:57:05,055 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 371 msec 2023-02-21 09:57:05,084 WARN [DataStreamer for file /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-02-21 09:57:05,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:05,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:05,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:05,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/.tmp/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:05,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/.tmp/cf/eaff9d6ee57e4a58a133e8233854e20f as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:05,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/.tmp/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:05,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f, entries=23, sequenceid=8, filesize=8.8 K 2023-02-21 09:57:05,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for dbcdd07ed31c76cf5ed03a249f4cf8d1 in 113ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:05,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/.tmp/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:05,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/.tmp/cf/2ffa7e7114804113a9806984d81b4e53 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:05,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:05,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:05,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:05,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:05,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/.tmp/cf/91b66786118a4ebf850b72bed26f0cae as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:05,161 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:05,161 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973425161"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973425161"}]},"ts":"1676973425161"} 2023-02-21 09:57:05,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53, entries=33, sequenceid=8, filesize=10.5 K 2023-02-21 09:57:05,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for a585cd7f02d60b76c310aa80c70f3a08 in 133ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:05,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=50 2023-02-21 09:57:05,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=50, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 473 msec 2023-02-21 09:57:05,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 493 msec 2023-02-21 09:57:05,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae, entries=21, sequenceid=8, filesize=8.5 K 2023-02-21 09:57:05,173 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:05,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.37 KB/1405, heapSize ~3.19 KB/3264, currentSize=0 B/0 for eb310106c5f65634673574312633a7b9 in 135ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:05,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:05,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:05,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:05,179 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:05,179 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425178"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973425178"}]},"ts":"1676973425178"} 2023-02-21 09:57:05,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:05,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:05,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:05,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:05,184 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=52 2023-02-21 09:57:05,184 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:05,184 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=52, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 498 msec 2023-02-21 09:57:05,184 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425184"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973425184"}]},"ts":"1676973425184"} 2023-02-21 09:57:05,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 508 msec 2023-02-21 09:57:05,188 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=51 2023-02-21 09:57:05,189 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=51, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 504 msec 2023-02-21 09:57:05,191 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=49 2023-02-21 09:57:05,191 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 513 msec 2023-02-21 09:57:05,192 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973425192"}]},"ts":"1676973425192"} 2023-02-21 09:57:05,194 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:05,196 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:05,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 531 msec 2023-02-21 09:57:05,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-21 09:57:05,425 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 49 completed 2023-02-21 09:57:05,426 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973425426 (current time:1676973425426). 2023-02-21 09:57:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x36eb826b to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:05,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dde82b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:05,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:05,435 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:05,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36eb826b to 127.0.0.1:58856 2023-02-21 09:57:05,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:05,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:05,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:05,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:05,440 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:05,441 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE; timeout=600000, timestamp=1676974025441 2023-02-21 09:57:05,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:05,441 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676973422348 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 2023-02-21 09:57:05,441 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:05,441 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:05,460 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,462 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,463 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,463 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,463 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,463 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:05,463 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,463 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:05,468 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758] hfiles 2023-02-21 09:57:05,468 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e] hfiles 2023-02-21 09:57:05,468 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:05,468 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f] hfiles 2023-02-21 09:57:05,468 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:05,468 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:05,469 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae] hfiles 2023-02-21 09:57:05,469 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:05,475 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720] hfiles 2023-02-21 09:57:05,476 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:05,477 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53] hfiles 2023-02-21 09:57:05,477 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:05,491 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388] hfiles 2023-02-21 09:57:05,491 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:05,491 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:05,491 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:05,492 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:05,492 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:05,492 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:05,514 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:05,514 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676973422348 2023-02-21 09:57:05,515 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973422348 2023-02-21 09:57:05,555 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973422348 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/snaptb0-1676973422348 2023-02-21 09:57:05,581 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676973422348 of table testRestoreSnapshot-1676973422348 completed 2023-02-21 09:57:05,581 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973422348 2023-02-21 09:57:05,582 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973422348 2023-02-21 09:57:05,582 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676973422348 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 at 1676973425441Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } at 1676973425460 (+19 ms)Consolidate snapshot: snaptb0-1676973422348 at 1676973425514 (+54 ms)Loading Region manifests for snaptb0-1676973422348 at 1676973425515 (+1 ms)Writing data manifest for snaptb0-1676973422348 at 1676973425529 (+14 ms)Verifying snapshot: snaptb0-1676973422348 at 1676973425539 (+10 ms)Snapshot snaptb0-1676973422348 of table testRestoreSnapshot-1676973422348 completed at 1676973425581 (+42 ms) 2023-02-21 09:57:05,583 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:05,585 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE in 145 msec 2023-02-21 09:57:05,691 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:05,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:05,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:05,693 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:05,694 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676973422348 2023-02-21 09:57:05,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:05,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-21 09:57:05,698 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676973422348 2023-02-21 09:57:05,698 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973425698"}]},"ts":"1676973425698"} 2023-02-21 09:57:05,700 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:05,705 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:05,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:05,707 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:05,708 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:05,708 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:05,708 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:05,708 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:05,709 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:05,709 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:05,709 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:05,709 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:05,709 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:05,709 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:05,709 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:05,709 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:05,709 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:05,709 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:05,710 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:05,711 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:05,711 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:05,859 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:05,859 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:05,859 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:05,860 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425859"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973425859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973425859"}]},"ts":"1676973425859"} 2023-02-21 09:57:05,859 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:05,860 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425859"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973425859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973425859"}]},"ts":"1676973425859"} 2023-02-21 09:57:05,860 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973425859"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973425859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973425859"}]},"ts":"1676973425859"} 2023-02-21 09:57:05,859 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:05,859 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:05,860 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425859"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973425859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973425859"}]},"ts":"1676973425859"} 2023-02-21 09:57:05,860 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973425859"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973425859"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973425859"}]},"ts":"1676973425859"} 2023-02-21 09:57:05,862 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=67, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:05,863 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=68, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:05,865 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=69, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:05,866 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=66, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:05,867 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=65, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:05,869 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:05,869 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973425868"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973425868"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973425868"}]},"ts":"1676973425868"} 2023-02-21 09:57:05,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=64, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:05,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-21 09:57:06,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:06,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:06,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:06,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:06,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,022 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,022 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,022 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:06,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:06,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:06,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,023 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:06,024 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:06,024 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:06,024 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:06,024 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:06,024 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,025 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:06,026 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:06,026 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:06,026 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:06,036 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:06,037 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:06,038 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:06,038 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:06,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,041 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:06,041 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:06,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:06,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:06,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59844080, jitterRate=-0.10825371742248535}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:06,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:06,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64278955, jitterRate=-0.04216893017292023}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:06,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:06,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:06,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=70, masterSystemTime=1676973426016 2023-02-21 09:57:06,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=71, masterSystemTime=1676973426017 2023-02-21 09:57:06,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74233334, jitterRate=0.10616287589073181}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:06,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:06,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=72, masterSystemTime=1676973426019 2023-02-21 09:57:06,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:06,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:06,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:06,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:06,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:06,052 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:06,052 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973426052"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973426052"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973426052"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973426052"}]},"ts":"1676973426052"} 2023-02-21 09:57:06,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:06,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:06,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:06,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:06,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:06,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,054 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:06,054 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973426054"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973426054"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973426054"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973426054"}]},"ts":"1676973426054"} 2023-02-21 09:57:06,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:06,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:06,055 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:06,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:06,055 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:06,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,056 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:06,056 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973426056"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973426056"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973426056"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973426056"}]},"ts":"1676973426056"} 2023-02-21 09:57:06,057 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:06,057 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:06,058 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,058 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,058 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:06,059 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=67 2023-02-21 09:57:06,059 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=67, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 193 msec 2023-02-21 09:57:06,059 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:06,060 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:06,060 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:06,060 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:06,060 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:06,060 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:06,061 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=68 2023-02-21 09:57:06,061 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=68, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 194 msec 2023-02-21 09:57:06,062 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 354 msec 2023-02-21 09:57:06,063 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=69 2023-02-21 09:57:06,063 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=69, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 194 msec 2023-02-21 09:57:06,064 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 356 msec 2023-02-21 09:57:06,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 358 msec 2023-02-21 09:57:06,071 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:06,071 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:06,071 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:06,071 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:06,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,075 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:06,075 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:06,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:06,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:06,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67918766, jitterRate=0.012068480253219604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:06,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:06,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60880773, jitterRate=-0.09280578792095184}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:06,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:06,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=74, masterSystemTime=1676973426016 2023-02-21 09:57:06,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=73, masterSystemTime=1676973426019 2023-02-21 09:57:06,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:06,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:06,083 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:06,083 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973426083"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973426083"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973426083"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973426083"}]},"ts":"1676973426083"} 2023-02-21 09:57:06,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:06,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:06,084 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:06,084 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973426084"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973426084"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973426084"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973426084"}]},"ts":"1676973426084"} 2023-02-21 09:57:06,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:06,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74842539, jitterRate=0.11524073779582977}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:06,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:06,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=75, masterSystemTime=1676973426017 2023-02-21 09:57:06,088 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=66 2023-02-21 09:57:06,088 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=66, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 219 msec 2023-02-21 09:57:06,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:06,089 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:06,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=65 2023-02-21 09:57:06,090 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=65, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 219 msec 2023-02-21 09:57:06,090 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:06,090 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973426090"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973426090"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973426090"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973426090"}]},"ts":"1676973426090"} 2023-02-21 09:57:06,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 383 msec 2023-02-21 09:57:06,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 384 msec 2023-02-21 09:57:06,094 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=64 2023-02-21 09:57:06,094 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=64, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 221 msec 2023-02-21 09:57:06,097 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-02-21 09:57:06,097 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 389 msec 2023-02-21 09:57:06,098 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973426098"}]},"ts":"1676973426098"} 2023-02-21 09:57:06,099 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:06,102 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676973422348' was successfully enabled. 2023-02-21 09:57:06,104 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676973422348 in 408 msec 2023-02-21 09:57:06,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-21 09:57:06,450 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 63 completed 2023-02-21 09:57:06,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:06,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:06,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:06,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:06,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:06,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:06,475 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676973422348 2023-02-21 09:57:06,475 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:06,477 INFO [Listener at localhost/40243] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-21 09:57:06,681 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:06,792 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:06,828 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-21 09:57:06,911 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-21 09:57:06,912 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-21 09:57:06,964 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:07,144 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:07,256 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:08,533 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-21 09:57:08,534 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-21 09:57:08,749 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(139): Connect 0x7042044c to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:08,756 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@700455dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:08,758 DEBUG [hconnection-0x6d06cc97-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:08,761 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37018, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:08,765 DEBUG [hconnection-0x6d06cc97-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:08,769 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37298, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:08,859 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:08,937 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:08,941 DEBUG [hconnection-0x6d06cc97-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:08,944 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43178, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:09,074 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:09,224 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:09,320 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:10,614 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:10,614 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:10,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:10,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-21 09:57:10,619 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973430619"}]},"ts":"1676973430619"} 2023-02-21 09:57:10,621 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:10,623 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:10,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:10,626 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:10,626 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:10,627 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:10,627 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:10,627 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:10,628 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:10,628 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:10,628 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973430628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973430628"}]},"ts":"1676973430628"} 2023-02-21 09:57:10,628 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:10,628 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:10,628 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973430628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973430628"}]},"ts":"1676973430628"} 2023-02-21 09:57:10,628 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973430628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973430628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973430628"}]},"ts":"1676973430628"} 2023-02-21 09:57:10,628 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973430628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973430628"}]},"ts":"1676973430628"} 2023-02-21 09:57:10,629 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:10,630 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973430629"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973430629"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973430629"}]},"ts":"1676973430629"} 2023-02-21 09:57:10,631 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=78, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:10,632 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=80, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:10,633 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=77, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:10,634 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=79, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:10,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:10,637 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:10,638 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:10,638 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430638"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973430638"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973430638"}]},"ts":"1676973430638"} 2023-02-21 09:57:10,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:10,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:10,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:10,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:10,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:10,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:10,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:10,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 74b647ed77adba33d8bf8ed03d9c05fc 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-21 09:57:10,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:10,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:10,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:10,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:10,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:10,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:10,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing dbcdd07ed31c76cf5ed03a249f4cf8d1 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-21 09:57:10,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:10,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:10,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:10,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:10,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:10,789 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:10,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a585cd7f02d60b76c310aa80c70f3a08 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-21 09:57:10,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 2023-02-21 09:57:10,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb 2023-02-21 09:57:10,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/.tmp/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:10,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/.tmp/cf/45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:10,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/.tmp/cf/45fb0f2a211b4901a56150a2b662707f as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:10,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/.tmp/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:10,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f, entries=35, sequenceid=15, filesize=10.9 K 2023-02-21 09:57:10,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59, entries=31, sequenceid=15, filesize=10.2 K 2023-02-21 09:57:10,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 74b647ed77adba33d8bf8ed03d9c05fc in 52ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:10,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for dbcdd07ed31c76cf5ed03a249f4cf8d1 in 51ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:10,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:10,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:10,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:10,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:10,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:10,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:10,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:10,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:10,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:10,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:10,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:10,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:10,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:10,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5c80767ec66fcdbcf22028d0c0cfeee5 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-21 09:57:10,851 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:10,851 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973430851"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973430851"}]},"ts":"1676973430851"} 2023-02-21 09:57:10,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:10,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:10,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:10,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:10,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:10,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:10,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:10,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing eb310106c5f65634673574312633a7b9 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-21 09:57:10,853 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:10,853 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973430853"}]},"ts":"1676973430853"} 2023-02-21 09:57:10,859 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=77 2023-02-21 09:57:10,860 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=77, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 222 msec 2023-02-21 09:57:10,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=80 2023-02-21 09:57:10,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=80, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 226 msec 2023-02-21 09:57:10,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 236 msec 2023-02-21 09:57:10,863 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 237 msec 2023-02-21 09:57:10,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-21 09:57:10,879 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 2023-02-21 09:57:10,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa 2023-02-21 09:57:10,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/.tmp/cf/b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:10,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/.tmp/cf/b4cba6164e5646ebb1df5c1417884979 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:10,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/.tmp/cf/a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:10,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979, entries=32, sequenceid=15, filesize=10.4 K 2023-02-21 09:57:10,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for eb310106c5f65634673574312633a7b9 in 44ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:10,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/.tmp/cf/a52c1f52ad274962a0a35c91207435f4 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:10,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:10,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:10,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:10,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4, entries=38, sequenceid=15, filesize=11.4 K 2023-02-21 09:57:10,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for 5c80767ec66fcdbcf22028d0c0cfeee5 in 58ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:10,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:10,910 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:10,911 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973430909"}]},"ts":"1676973430909"} 2023-02-21 09:57:10,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:10,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=78 2023-02-21 09:57:10,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=78, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 281 msec 2023-02-21 09:57:10,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:10,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:10,916 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 290 msec 2023-02-21 09:57:10,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:10,917 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:10,917 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973430917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973430917"}]},"ts":"1676973430917"} 2023-02-21 09:57:10,920 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-21 09:57:10,920 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 279 msec 2023-02-21 09:57:10,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 296 msec 2023-02-21 09:57:11,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c 2023-02-21 09:57:11,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/.tmp/cf/a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:11,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/.tmp/cf/a912b8ec2a324a91924827dcf86ef9ec as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:11,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec, entries=37, sequenceid=15, filesize=11.2 K 2023-02-21 09:57:11,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for a585cd7f02d60b76c310aa80c70f3a08 in 454ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:11,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:11,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:11,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:11,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:11,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:11,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:11,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:11,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:11,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:11,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:11,254 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:11,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5863e48176a446e50692d5bcf8d380f1 1/1 column families, dataSize=21.39 KB heapSize=46.23 KB 2023-02-21 09:57:11,255 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973431254"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973431254"}]},"ts":"1676973431254"} 2023-02-21 09:57:11,259 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=79 2023-02-21 09:57:11,259 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=79, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 623 msec 2023-02-21 09:57:11,265 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 635 msec 2023-02-21 09:57:11,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e 2023-02-21 09:57:11,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/.tmp/cf/5d93932811904d658444a38dedb905b4 2023-02-21 09:57:11,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/.tmp/cf/5d93932811904d658444a38dedb905b4 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4 2023-02-21 09:57:11,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4, entries=327, sequenceid=15, filesize=60.6 K 2023-02-21 09:57:11,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.39 KB/21905, heapSize ~46.22 KB/47328, currentSize=0 B/0 for 5863e48176a446e50692d5bcf8d380f1 in 42ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:11,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:11,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:11,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:11,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:11,308 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:11,308 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973431308"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973431308"}]},"ts":"1676973431308"} 2023-02-21 09:57:11,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-21 09:57:11,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 674 msec 2023-02-21 09:57:11,313 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=76 2023-02-21 09:57:11,313 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 687 msec 2023-02-21 09:57:11,314 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973431314"}]},"ts":"1676973431314"} 2023-02-21 09:57:11,315 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:11,317 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:11,319 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 704 msec 2023-02-21 09:57:11,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-21 09:57:11,372 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 76 completed 2023-02-21 09:57:11,373 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973431373 (current time:1676973431373). 2023-02-21 09:57:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x3dd04b0b to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:11,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57c27e00, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:11,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:11,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37032, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:11,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dd04b0b to 127.0.0.1:58856 2023-02-21 09:57:11,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:11,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:11,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:11,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:11,386 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:11,387 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE; timeout=600000, timestamp=1676974031387 2023-02-21 09:57:11,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:11,387 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676973422348 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 2023-02-21 09:57:11,387 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:11,387 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:11,396 WARN [DataStreamer for file /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb1-1676973422348/.snapshotinfo] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-02-21 09:57:11,400 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:11,402 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,402 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,402 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,402 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,402 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,402 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,403 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,403 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,404 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,404 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,404 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,404 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4] hfiles 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720] hfiles 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979] hfiles 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:11,407 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:11,409 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,410 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,413 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4] hfiles 2023-02-21 09:57:11,413 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:11,413 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4 2023-02-21 09:57:11,413 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f] hfiles 2023-02-21 09:57:11,413 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:11,414 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:11,414 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec] hfiles 2023-02-21 09:57:11,414 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:11,414 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388] hfiles 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 2023-02-21 09:57:11,415 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:11,450 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:11,450 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676973422348 2023-02-21 09:57:11,451 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb1-1676973422348 2023-02-21 09:57:11,482 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb1-1676973422348 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/snaptb1-1676973422348 2023-02-21 09:57:11,503 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676973422348 of table testRestoreSnapshot-1676973422348 completed 2023-02-21 09:57:11,503 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb1-1676973422348 2023-02-21 09:57:11,503 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb1-1676973422348 2023-02-21 09:57:11,503 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676973422348 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 at 1676973431387Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } at 1676973431400 (+13 ms)Consolidate snapshot: snaptb1-1676973422348 at 1676973431450 (+50 ms)Loading Region manifests for snaptb1-1676973422348 at 1676973431450Writing data manifest for snaptb1-1676973422348 at 1676973431455 (+5 ms)Verifying snapshot: snaptb1-1676973422348 at 1676973431469 (+14 ms)Snapshot snaptb1-1676973422348 of table testRestoreSnapshot-1676973422348 completed at 1676973431503 (+34 ms) 2023-02-21 09:57:11,505 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:11,506 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE in 121 msec 2023-02-21 09:57:11,638 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:11,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:11,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:11,655 INFO [Listener at localhost/40243] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:11,656 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:11,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973431656 (current time:1676973431656). 2023-02-21 09:57:11,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:11,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x5cd54b69 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:11,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1da67464, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:11,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:11,665 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37034, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5cd54b69 to 127.0.0.1:58856 2023-02-21 09:57:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:11,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:11,672 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:11,673 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE; timeout=600000, timestamp=1676974031673 2023-02-21 09:57:11,673 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676973422348-1676973431655 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 2023-02-21 09:57:11,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:11,674 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:11,674 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:11,690 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,692 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,693 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,693 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,693 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:11,693 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:11,696 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720] hfiles 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec] hfiles 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f] hfiles 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979] hfiles 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4] hfiles 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4] hfiles 2023-02-21 09:57:11,697 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:11,698 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:11,698 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:11,698 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4 2023-02-21 09:57:11,698 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:11,698 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:11,698 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388] hfiles 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 2023-02-21 09:57:11,711 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:11,758 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7042044c to 127.0.0.1:58856 2023-02-21 09:57:11,758 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:11,761 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:11,761 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:11,762 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:11,794 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676973422348-1676973431655 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:11,816 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676973422348-1676973431655 of table testRestoreSnapshot-1676973422348 completed 2023-02-21 09:57:11,816 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:11,816 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:11,817 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676973422348-1676973431655 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 at 1676973431673Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } at 1676973431690 (+17 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676973422348-1676973431655 at 1676973431761 (+71 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676973422348-1676973431655 at 1676973431761Writing data manifest for hbase-failsafe-snaptb0-1676973422348-1676973431655 at 1676973431765 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676973422348-1676973431655 at 1676973431781 (+16 ms)Snapshot hbase-failsafe-snaptb0-1676973422348-1676973431655 of table testRestoreSnapshot-1676973422348 completed at 1676973431816 (+35 ms) 2023-02-21 09:57:11,818 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:11,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE in 150 msec 2023-02-21 09:57:11,924 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:11,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:11,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676973422348-1676973431655 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "snaptb0-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973425426 type: DISABLED version: 2 ttl: 0 ) 2023-02-21 09:57:11,954 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676973422348 as table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-21 09:57:11,972 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000002 2023-02-21 09:57:11,973 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:11,973 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000002 2023-02-21 09:57:11,974 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:11,978 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973425426 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:11,979 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348 2023-02-21 09:57:11,988 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:11,996 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa 2023-02-21 09:57:11,998 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 2023-02-21 09:57:12,000 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 2023-02-21 09:57:12,001 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c 2023-02-21 09:57:12,002 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e 2023-02-21 09:57:12,004 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb 2023-02-21 09:57:12,004 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,004 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,004 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,004 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,004 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,004 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,009 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:12,010 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/5d93932811904d658444a38dedb905b4 2023-02-21 09:57:12,010 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:12,010 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:12,010 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:12,011 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:12,011 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973425426 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:12,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973432013"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:12,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973432013"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:12,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973432013"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:12,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973432013"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:12,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973432013"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:12,013 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973432013"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:12,017 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973432014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973432014"}]},"ts":"1676973432014"} 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973432014"}]},"ts":"1676973432014"} 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973432014"}]},"ts":"1676973432014"} 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973432014"}]},"ts":"1676973432014"} 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973432014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973432014"}]},"ts":"1676973432014"} 2023-02-21 09:57:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973432014"}]},"ts":"1676973432014"} 2023-02-21 09:57:12,020 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-21 09:57:12,020 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-21 09:57:12,020 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-21 09:57:12,020 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676973422348 completed! 2023-02-21 09:57:12,023 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "snaptb0-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973425426 type: DISABLED version: 2 ttl: 0 ) in 76 msec 2023-02-21 09:57:12,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-21 09:57:12,209 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676973422348, procId: 91 completed 2023-02-21 09:57:12,209 INFO [Listener at localhost/40243] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:12,214 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676973422348-1676973431655" 2023-02-21 09:57:12,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676973422348-1676973431655 2023-02-21 09:57:12,218 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:12,218 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676973422348 2023-02-21 09:57:12,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:12,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-21 09:57:12,223 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676973422348 2023-02-21 09:57:12,223 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973432223"}]},"ts":"1676973432223"} 2023-02-21 09:57:12,224 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:12,228 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:12,230 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:12,232 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:12,232 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:12,232 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:12,233 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:12,233 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:12,233 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:12,233 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:12,233 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:12,234 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:12,234 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:12,234 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:12,235 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:12,383 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:12,384 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:12,384 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:12,384 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:12,384 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:12,387 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:12,387 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:12,387 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973432387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973432387"}]},"ts":"1676973432387"} 2023-02-21 09:57:12,387 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:12,387 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:12,387 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:12,388 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973432387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973432387"}]},"ts":"1676973432387"} 2023-02-21 09:57:12,387 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973432387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973432387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973432387"}]},"ts":"1676973432387"} 2023-02-21 09:57:12,387 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973432387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973432387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973432387"}]},"ts":"1676973432387"} 2023-02-21 09:57:12,388 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973432387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973432387"}]},"ts":"1676973432387"} 2023-02-21 09:57:12,389 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=96, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:12,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:12,391 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=93, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:12,392 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=98, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:12,393 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:12,393 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432393"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973432393"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973432393"}]},"ts":"1676973432393"} 2023-02-21 09:57:12,393 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=94, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:12,395 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:12,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-21 09:57:12,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:12,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:12,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,546 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,546 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,547 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:12,547 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:12,547 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:12,547 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:12,547 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:12,547 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:12,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:12,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:12,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:12,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,552 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,553 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:12,553 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:12,553 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:12,553 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:12,553 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:12,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,555 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:12,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:12,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70133847, jitterRate=0.045075759291648865}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:12,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:12,563 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:12,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=99, masterSystemTime=1676973432541 2023-02-21 09:57:12,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:12,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:12,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:12,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:12,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,567 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:12,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:12,567 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:12,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,567 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:12,567 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432567"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973432567"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973432567"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973432567"}]},"ts":"1676973432567"} 2023-02-21 09:57:12,568 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:12,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70547382, jitterRate=0.051237910985946655}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:12,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:12,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=100, masterSystemTime=1676973432542 2023-02-21 09:57:12,572 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:12,572 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:12,572 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:12,573 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=96 2023-02-21 09:57:12,573 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=96, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584 in 181 msec 2023-02-21 09:57:12,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:12,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:12,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:12,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:12,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:12,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:12,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,574 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:12,574 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432574"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973432574"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973432574"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973432574"}]},"ts":"1676973432574"} 2023-02-21 09:57:12,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62113672, jitterRate=-0.07443416118621826}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:12,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:12,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 343 msec 2023-02-21 09:57:12,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=104, masterSystemTime=1676973432548 2023-02-21 09:57:12,576 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,577 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:12,578 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:12,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:12,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:12,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:12,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:12,579 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:12,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:12,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,579 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:12,579 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432579"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973432579"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973432579"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973432579"}]},"ts":"1676973432579"} 2023-02-21 09:57:12,580 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,580 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:12,581 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:12,581 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-02-21 09:57:12,581 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539 in 187 msec 2023-02-21 09:57:12,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,582 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:12,582 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:12,582 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:12,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 351 msec 2023-02-21 09:57:12,584 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-21 09:57:12,585 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614 in 186 msec 2023-02-21 09:57:12,586 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 355 msec 2023-02-21 09:57:12,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:12,587 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:12,587 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:12,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69023459, jitterRate=0.028529688715934753}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:12,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:12,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,589 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:12,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=101, masterSystemTime=1676973432541 2023-02-21 09:57:12,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,589 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:12,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:12,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:12,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,591 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:12,592 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973432591"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973432591"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973432591"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973432591"}]},"ts":"1676973432591"} 2023-02-21 09:57:12,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:12,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69840982, jitterRate=0.04071173071861267}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:12,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:12,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=103, masterSystemTime=1676973432542 2023-02-21 09:57:12,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:12,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60199430, jitterRate=-0.10295858979225159}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:12,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:12,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=93 2023-02-21 09:57:12,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=93, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 202 msec 2023-02-21 09:57:12,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:12,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:12,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=102, masterSystemTime=1676973432548 2023-02-21 09:57:12,596 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:12,596 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973432596"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973432596"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973432596"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973432596"}]},"ts":"1676973432596"} 2023-02-21 09:57:12,597 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 367 msec 2023-02-21 09:57:12,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:12,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:12,598 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:12,598 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973432598"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973432598"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973432598"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973432598"}]},"ts":"1676973432598"} 2023-02-21 09:57:12,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=94 2023-02-21 09:57:12,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=94, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 204 msec 2023-02-21 09:57:12,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 372 msec 2023-02-21 09:57:12,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=98 2023-02-21 09:57:12,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=98, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 207 msec 2023-02-21 09:57:12,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=98, resume processing ppid=92 2023-02-21 09:57:12,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 371 msec 2023-02-21 09:57:12,603 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973432603"}]},"ts":"1676973432603"} 2023-02-21 09:57:12,605 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:12,606 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676973422348' was successfully enabled. 2023-02-21 09:57:12,608 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676973422348 in 388 msec 2023-02-21 09:57:12,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-21 09:57:12,974 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 92 completed 2023-02-21 09:57:12,975 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(139): Connect 0x1b384ceb to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:12,981 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@aebe9d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:12,982 DEBUG [hconnection-0x272618c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:12,986 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37040, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:12,990 DEBUG [hconnection-0x272618c8-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:12,992 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43182, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:13,064 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:13,131 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:13,134 DEBUG [hconnection-0x272618c8-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:13,137 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37308, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:13,242 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:13,359 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:13,438 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:14,431 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:14,431 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:14,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:14,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-21 09:57:14,435 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973434435"}]},"ts":"1676973434435"} 2023-02-21 09:57:14,436 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:14,438 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:14,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:14,439 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:14,440 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:14,441 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:14,441 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:14,441 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:14,441 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:14,441 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:14,442 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:14,442 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:14,442 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:14,442 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434442"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973434442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973434442"}]},"ts":"1676973434442"} 2023-02-21 09:57:14,442 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434441"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973434441"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973434441"}]},"ts":"1676973434441"} 2023-02-21 09:57:14,442 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434441"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973434441"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973434441"}]},"ts":"1676973434441"} 2023-02-21 09:57:14,442 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434441"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973434441"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973434441"}]},"ts":"1676973434441"} 2023-02-21 09:57:14,442 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973434441"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973434441"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973434441"}]},"ts":"1676973434441"} 2023-02-21 09:57:14,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=107, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:14,446 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=108, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:14,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=110, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:14,448 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:14,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=111, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:14,451 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:14,452 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:14,453 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973434452"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973434452"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973434452"}]},"ts":"1676973434452"} 2023-02-21 09:57:14,454 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=106, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:14,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:14,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:14,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:14,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:14,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:14,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:14,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:14,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:14,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:14,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:14,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:14,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:14,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:14,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:14,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:14,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:14,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:14,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:14,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-21 09:57:14,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-21 09:57:14,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-21 09:57:14,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:14,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:14,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:14,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:14,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:14,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:14,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:14,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:14,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:14,615 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:14,615 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434615"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973434615"}]},"ts":"1676973434615"} 2023-02-21 09:57:14,615 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:14,615 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:14,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:14,615 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:14,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:14,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:14,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:14,616 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:14,616 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973434616"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973434616"}]},"ts":"1676973434616"} 2023-02-21 09:57:14,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:14,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:14,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:14,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:14,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:14,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:14,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:14,617 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:14,617 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973434617"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973434617"}]},"ts":"1676973434617"} 2023-02-21 09:57:14,621 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=107 2023-02-21 09:57:14,621 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=107, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 174 msec 2023-02-21 09:57:14,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=106 2023-02-21 09:57:14,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=106, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 165 msec 2023-02-21 09:57:14,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-21 09:57:14,622 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=111 2023-02-21 09:57:14,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-21 09:57:14,622 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=111, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 170 msec 2023-02-21 09:57:14,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:14,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:14,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 182 msec 2023-02-21 09:57:14,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 182 msec 2023-02-21 09:57:14,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:14,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:14,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 183 msec 2023-02-21 09:57:14,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:14,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-21 09:57:14,625 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:14,625 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434625"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973434625"}]},"ts":"1676973434625"} 2023-02-21 09:57:14,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:14,625 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:14,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:14,626 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434625"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973434625"}]},"ts":"1676973434625"} 2023-02-21 09:57:14,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:14,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:14,627 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:14,627 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973434627"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973434627"}]},"ts":"1676973434627"} 2023-02-21 09:57:14,628 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-21 09:57:14,628 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584 in 178 msec 2023-02-21 09:57:14,629 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=110 2023-02-21 09:57:14,629 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=110, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539 in 180 msec 2023-02-21 09:57:14,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 189 msec 2023-02-21 09:57:14,630 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 190 msec 2023-02-21 09:57:14,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=108 2023-02-21 09:57:14,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=108, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614 in 185 msec 2023-02-21 09:57:14,631 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=108, resume processing ppid=105 2023-02-21 09:57:14,631 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 191 msec 2023-02-21 09:57:14,632 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973434632"}]},"ts":"1676973434632"} 2023-02-21 09:57:14,633 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:14,635 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:14,636 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 204 msec 2023-02-21 09:57:14,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-21 09:57:14,686 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 105 completed 2023-02-21 09:57:14,694 INFO [Listener at localhost/40243] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:14,695 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:14,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973434695 (current time:1676973434695). 2023-02-21 09:57:14,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:14,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x1711be4e to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:14,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2aff341e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:14,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:14,704 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38348, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:14,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1711be4e to 127.0.0.1:58856 2023-02-21 09:57:14,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:14,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:14,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:14,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:14,708 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:14,708 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE; timeout=600000, timestamp=1676974034708 2023-02-21 09:57:14,708 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676973422348-1676973434694 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 2023-02-21 09:57:14,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:14,709 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:14,709 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:14,718 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:14,723 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,723 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,723 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,723 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:14,724 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:14,728 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720] hfiles 2023-02-21 09:57:14,728 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53] hfiles 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e] hfiles 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae] hfiles 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758] hfiles 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:14,728 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f] hfiles 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:14,729 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388] hfiles 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:14,741 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:14,959 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:14,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:14,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' is still in progress! 2023-02-21 09:57:14,960 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-21 09:57:15,160 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:15,161 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,161 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,188 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676973422348-1676973434694 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,209 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676973422348-1676973434694 of table testRestoreSnapshot-1676973422348 completed 2023-02-21 09:57:15,209 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,209 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,209 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676973422348-1676973434694 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 at 1676973434708Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } at 1676973434718 (+10 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676973422348-1676973434694 at 1676973435161 (+443 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676973422348-1676973434694 at 1676973435161Writing data manifest for hbase-failsafe-emptySnaptb-1676973422348-1676973434694 at 1676973435168 (+7 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676973422348-1676973434694 at 1676973435181 (+13 ms)Snapshot hbase-failsafe-emptySnaptb-1676973422348-1676973434694 of table testRestoreSnapshot-1676973422348 completed at 1676973435209 (+28 ms) 2023-02-21 09:57:15,211 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:15,212 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE in 505 msec 2023-02-21 09:57:15,460 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:15,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:15,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676973422348-1676973434694 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:15,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "emptySnaptb-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973423483 type: DISABLED version: 2 ttl: 0 ) 2023-02-21 09:57:15,468 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676973422348 as table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-21 09:57:15,485 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000003 2023-02-21 09:57:15,486 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000002 2023-02-21 09:57:15,486 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000003 2023-02-21 09:57:15,487 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:15,488 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973423483 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:15,488 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,495 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,499 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:15,500 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:15,501 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:15,503 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:15,505 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:15,506 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:15,506 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:15,506 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:15,506 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:15,506 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:15,506 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:15,506 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: eb310106c5f65634673574312633a7b9 2023-02-21 09:57:15,512 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:15,512 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:15,512 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:15,512 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:15,513 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/d65bd76661554d98a17951ff943be720 2023-02-21 09:57:15,514 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:15,514 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973423483 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:15,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973435515"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:15,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973435515"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:15,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973435515"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:15,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973435515"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:15,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973435515"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:15,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973435515"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:15,518 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-21 09:57:15,518 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-21 09:57:15,518 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973435516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973435516"}]},"ts":"1676973435516"} 2023-02-21 09:57:15,519 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973435516"}]},"ts":"1676973435516"} 2023-02-21 09:57:15,519 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973435516"}]},"ts":"1676973435516"} 2023-02-21 09:57:15,519 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973435516"}]},"ts":"1676973435516"} 2023-02-21 09:57:15,519 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973435516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973435516"}]},"ts":"1676973435516"} 2023-02-21 09:57:15,519 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973435516"}]},"ts":"1676973435516"} 2023-02-21 09:57:15,521 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-21 09:57:15,521 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-21 09:57:15,521 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-21 09:57:15,521 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676973422348 completed! 2023-02-21 09:57:15,523 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "emptySnaptb-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973423483 type: DISABLED version: 2 ttl: 0 ) in 55 msec 2023-02-21 09:57:15,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-21 09:57:15,721 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676973422348, procId: 119 completed 2023-02-21 09:57:15,721 INFO [Listener at localhost/40243] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,722 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676973422348-1676973434694" 2023-02-21 09:57:15,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676973422348-1676973434694 2023-02-21 09:57:15,725 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,725 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-21 09:57:15,729 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676973422348 2023-02-21 09:57:15,729 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973435729"}]},"ts":"1676973435729"} 2023-02-21 09:57:15,730 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:15,733 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:15,733 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:15,736 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:15,736 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:15,736 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:15,736 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:15,736 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:15,737 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:15,737 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:15,737 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:15,737 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:15,737 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:15,738 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:15,738 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:15,887 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:15,887 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:15,887 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:15,887 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:15,887 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:15,892 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:15,892 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:15,892 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:15,892 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973435892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973435892"}]},"ts":"1676973435892"} 2023-02-21 09:57:15,892 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973435892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973435892"}]},"ts":"1676973435892"} 2023-02-21 09:57:15,892 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:15,892 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:15,892 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973435892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973435892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973435892"}]},"ts":"1676973435892"} 2023-02-21 09:57:15,892 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973435892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973435892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973435892"}]},"ts":"1676973435892"} 2023-02-21 09:57:15,892 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973435892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973435892"}]},"ts":"1676973435892"} 2023-02-21 09:57:15,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=124, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:15,894 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=123, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:15,895 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=126, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:15,896 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=121, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:15,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=125, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:15,897 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:15,897 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973435897"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973435897"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973435897"}]},"ts":"1676973435897"} 2023-02-21 09:57:15,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=122, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:15,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-21 09:57:16,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:16,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:16,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:16,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:16,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,049 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,050 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,051 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:16,051 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:16,051 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:16,051 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:16,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,051 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:16,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:16,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,052 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:16,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:16,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,053 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:16,053 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:16,053 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,054 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:16,054 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:16,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,055 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:16,055 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:16,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61463203, jitterRate=-0.08412690460681915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:16,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60423712, jitterRate=-0.09961652755737305}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:16,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:16,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:16,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=129, masterSystemTime=1676973436046 2023-02-21 09:57:16,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=130, masterSystemTime=1676973436045 2023-02-21 09:57:16,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65874374, jitterRate=-0.018395334482192993}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:16,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:16,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:16,060 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,061 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973436060"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973436060"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973436060"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973436060"}]},"ts":"1676973436060"} 2023-02-21 09:57:16,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:16,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=132, masterSystemTime=1676973436049 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:16,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:16,062 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:16,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,062 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973436062"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973436062"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973436062"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973436062"}]},"ts":"1676973436062"} 2023-02-21 09:57:16,062 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,063 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,064 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:16,064 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:16,064 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:16,064 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:16,064 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:16,065 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:16,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=126 2023-02-21 09:57:16,065 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:16,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=126, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 168 msec 2023-02-21 09:57:16,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:16,066 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:16,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,066 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:16,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:16,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,066 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436065"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973436065"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973436065"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973436065"}]},"ts":"1676973436065"} 2023-02-21 09:57:16,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,070 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=121 2023-02-21 09:57:16,070 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=121, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,46077,1676973419539 in 168 msec 2023-02-21 09:57:16,070 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 332 msec 2023-02-21 09:57:16,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,071 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 337 msec 2023-02-21 09:57:16,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65030269, jitterRate=-0.03097347915172577}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:16,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=122 2023-02-21 09:57:16,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:16,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=122, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,41011,1676973419614 in 171 msec 2023-02-21 09:57:16,073 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=128, masterSystemTime=1676973436046 2023-02-21 09:57:16,074 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 339 msec 2023-02-21 09:57:16,074 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:16,074 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:16,074 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:16,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63730095, jitterRate=-0.05034758150577545}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:16,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:16,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=127, masterSystemTime=1676973436045 2023-02-21 09:57:16,075 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:16,075 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:16,075 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436075"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973436075"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973436075"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973436075"}]},"ts":"1676973436075"} 2023-02-21 09:57:16,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,077 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:16,077 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436077"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973436077"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973436077"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973436077"}]},"ts":"1676973436077"} 2023-02-21 09:57:16,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,078 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=123 2023-02-21 09:57:16,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=123, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 183 msec 2023-02-21 09:57:16,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 346 msec 2023-02-21 09:57:16,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=124 2023-02-21 09:57:16,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=124, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 186 msec 2023-02-21 09:57:16,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 348 msec 2023-02-21 09:57:16,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71976447, jitterRate=0.07253263890743256}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:16,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:16,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=131, masterSystemTime=1676973436049 2023-02-21 09:57:16,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,084 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:16,084 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436084"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973436084"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973436084"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973436084"}]},"ts":"1676973436084"} 2023-02-21 09:57:16,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=125 2023-02-21 09:57:16,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=125, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 188 msec 2023-02-21 09:57:16,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=125, resume processing ppid=120 2023-02-21 09:57:16,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 354 msec 2023-02-21 09:57:16,088 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973436088"}]},"ts":"1676973436088"} 2023-02-21 09:57:16,089 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:16,091 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676973422348' was successfully enabled. 2023-02-21 09:57:16,093 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676973422348 in 367 msec 2023-02-21 09:57:16,350 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-21 09:57:16,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-21 09:57:16,480 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 120 completed 2023-02-21 09:57:16,480 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(139): Connect 0x2c2934c9 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:16,486 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f3a7cee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:16,488 DEBUG [hconnection-0x43ec0260-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:16,490 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38358, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:16,494 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:16,496 DEBUG [hconnection-0x43ec0260-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:16,498 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56152, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:16,499 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:16,501 DEBUG [hconnection-0x43ec0260-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:16,505 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39830, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:16,505 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:16,508 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:16,511 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:16,516 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:16,516 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:16,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:16,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-21 09:57:16,519 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973436519"}]},"ts":"1676973436519"} 2023-02-21 09:57:16,520 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:16,523 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:16,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:16,525 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:16,525 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:16,525 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:16,525 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:16,525 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:16,526 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:16,526 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436526"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973436526"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973436526"}]},"ts":"1676973436526"} 2023-02-21 09:57:16,526 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:16,526 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:16,526 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973436526"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973436526"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973436526"}]},"ts":"1676973436526"} 2023-02-21 09:57:16,526 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436526"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973436526"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973436526"}]},"ts":"1676973436526"} 2023-02-21 09:57:16,527 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:16,527 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:16,527 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436526"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973436526"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973436526"}]},"ts":"1676973436526"} 2023-02-21 09:57:16,527 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436526"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973436526"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973436526"}]},"ts":"1676973436526"} 2023-02-21 09:57:16,527 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=137, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:16,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=139, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:16,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=138, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:16,529 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=136, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:16,530 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:16,530 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=135, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:16,531 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:16,531 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973436531"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973436531"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973436531"}]},"ts":"1676973436531"} 2023-02-21 09:57:16,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:16,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:16,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:16,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:16,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:16,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:16,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:16,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-21 09:57:16,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-21 09:57:16,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-21 09:57:16,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:16,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:16,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:16,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:16,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:16,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:16,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:16,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:16,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:16,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,690 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:16,690 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436690"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973436690"}]},"ts":"1676973436690"} 2023-02-21 09:57:16,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:16,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:16,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:16,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,692 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:16,692 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973436692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973436692"}]},"ts":"1676973436692"} 2023-02-21 09:57:16,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:16,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:16,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:16,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-21 09:57:16,695 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:16,695 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436695"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973436695"}]},"ts":"1676973436695"} 2023-02-21 09:57:16,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=135 2023-02-21 09:57:16,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-21 09:57:16,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=135, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,41011,1676973419614 in 165 msec 2023-02-21 09:57:16,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-21 09:57:16,699 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-02-21 09:57:16,699 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,46077,1676973419539 in 164 msec 2023-02-21 09:57:16,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:16,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:16,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:16,700 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 174 msec 2023-02-21 09:57:16,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:16,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:16,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:16,701 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=136 2023-02-21 09:57:16,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 175 msec 2023-02-21 09:57:16,701 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=136, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,44835,1676973419584 in 168 msec 2023-02-21 09:57:16,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:16,702 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 177 msec 2023-02-21 09:57:16,702 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:16,702 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973436702"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973436702"}]},"ts":"1676973436702"} 2023-02-21 09:57:16,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:16,703 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:16,703 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436703"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973436703"}]},"ts":"1676973436703"} 2023-02-21 09:57:16,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:16,704 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:16,704 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973436704"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973436704"}]},"ts":"1676973436704"} 2023-02-21 09:57:16,708 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=139 2023-02-21 09:57:16,708 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=139, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 176 msec 2023-02-21 09:57:16,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=138 2023-02-21 09:57:16,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=137 2023-02-21 09:57:16,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=138, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 175 msec 2023-02-21 09:57:16,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=137, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,46077,1676973419539 in 178 msec 2023-02-21 09:57:16,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 184 msec 2023-02-21 09:57:16,710 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 185 msec 2023-02-21 09:57:16,710 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=137, resume processing ppid=133 2023-02-21 09:57:16,710 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 185 msec 2023-02-21 09:57:16,711 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973436711"}]},"ts":"1676973436711"} 2023-02-21 09:57:16,716 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:16,718 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:16,720 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 202 msec 2023-02-21 09:57:16,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-21 09:57:16,770 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 133 completed 2023-02-21 09:57:16,779 INFO [Listener at localhost/40243] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:16,780 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:16,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973436780 (current time:1676973436780). 2023-02-21 09:57:16,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:16,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x21bed822 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31bb2c10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:16,788 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38366, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:16,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21bed822 to 127.0.0.1:58856 2023-02-21 09:57:16,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:16,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:16,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:16,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:16,792 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:16,793 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE; timeout=600000, timestamp=1676974036793 2023-02-21 09:57:16,793 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676973422348-1676973436779 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 2023-02-21 09:57:16,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } 2023-02-21 09:57:16,793 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:16,793 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:16,802 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,806 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,807 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,807 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,807 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,807 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,808 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:16,828 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:16,828 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:16,829 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:16,845 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676973422348-1676973436779 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:16,862 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676973422348-1676973436779 of table testRestoreSnapshot-1676973422348 completed 2023-02-21 09:57:16,862 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:16,862 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:16,862 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676973422348-1676973436779 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676973422348 at 1676973436793Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } at 1676973436802 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676973422348-1676973436779 at 1676973436828 (+26 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676973422348-1676973436779 at 1676973436828Writing data manifest for hbase-failsafe-snaptb1-1676973422348-1676973436779 at 1676973436831 (+3 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676973422348-1676973436779 at 1676973436840 (+9 ms)Snapshot hbase-failsafe-snaptb1-1676973422348-1676973436779 of table testRestoreSnapshot-1676973422348 completed at 1676973436862 (+22 ms) 2023-02-21 09:57:16,864 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE 2023-02-21 09:57:16,865 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676973422348, type=EXCLUSIVE in 73 msec 2023-02-21 09:57:17,044 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:17,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 } is done 2023-02-21 09:57:17,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676973422348-1676973436779 table=testRestoreSnapshot-1676973422348 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 ) 2023-02-21 09:57:17,066 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676973422348 as table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-21 09:57:17,096 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000004 2023-02-21 09:57:17,097 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000003 2023-02-21 09:57:17,097 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000004 2023-02-21 09:57:17,098 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } 2023-02-21 09:57:17,101 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:17,101 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,109 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,111 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,114 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,118 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,121 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,125 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,128 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,131 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,135 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,139 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,144 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,148 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,155 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 of snapshot=snaptb1-1676973422348 to region=9cd55b007e40a5b23f7970090b429650 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,159 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,160 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,160 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,160 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,160 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,160 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,169 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 1d0a8ea5761f44caa7dda024184b0758 of snapshot=snaptb1-1676973422348 to region=5863e48176a446e50692d5bcf8d380f1 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,169 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2d39b975032e413ab7e72e6533951b3e of snapshot=snaptb1-1676973422348 to region=5c80767ec66fcdbcf22028d0c0cfeee5 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,169 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 45fb0f2a211b4901a56150a2b662707f of snapshot=snaptb1-1676973422348 to region=74b647ed77adba33d8bf8ed03d9c05fc table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,171 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2ffa7e7114804113a9806984d81b4e53 of snapshot=snaptb1-1676973422348 to region=a585cd7f02d60b76c310aa80c70f3a08 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,171 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 03e1eeaaf8564145a9dc0fa8a88a2d59 of snapshot=snaptb1-1676973422348 to region=dbcdd07ed31c76cf5ed03a249f4cf8d1 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,172 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 91b66786118a4ebf850b72bed26f0cae of snapshot=snaptb1-1676973422348 to region=eb310106c5f65634673574312633a7b9 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,179 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a52c1f52ad274962a0a35c91207435f4 of snapshot=snaptb1-1676973422348 to region=5c80767ec66fcdbcf22028d0c0cfeee5 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,180 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d65bd76661554d98a17951ff943be720 of snapshot=snaptb1-1676973422348 to region=74b647ed77adba33d8bf8ed03d9c05fc table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,180 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a912b8ec2a324a91924827dcf86ef9ec of snapshot=snaptb1-1676973422348 to region=a585cd7f02d60b76c310aa80c70f3a08 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,181 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 5d93932811904d658444a38dedb905b4 of snapshot=snaptb1-1676973422348 to region=5863e48176a446e50692d5bcf8d380f1 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,184 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eaff9d6ee57e4a58a133e8233854e20f of snapshot=snaptb1-1676973422348 to region=dbcdd07ed31c76cf5ed03a249f4cf8d1 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,184 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink b4cba6164e5646ebb1df5c1417884979 of snapshot=snaptb1-1676973422348 to region=eb310106c5f65634673574312633a7b9 table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,190 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:17,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973437191"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:17,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973437191"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:17,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973437191"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:17,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973437191"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:17,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973437191"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:17,191 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973437191"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:17,193 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-21 09:57:17,193 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-21 09:57:17,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973437192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973437192"}]},"ts":"1676973437192"} 2023-02-21 09:57:17,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973437192"}]},"ts":"1676973437192"} 2023-02-21 09:57:17,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973437192"}]},"ts":"1676973437192"} 2023-02-21 09:57:17,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973437192"}]},"ts":"1676973437192"} 2023-02-21 09:57:17,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973437192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973437192"}]},"ts":"1676973437192"} 2023-02-21 09:57:17,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973437192"}]},"ts":"1676973437192"} 2023-02-21 09:57:17,195 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-21 09:57:17,196 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-21 09:57:17,196 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-21 09:57:17,196 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676973422348 table=testRestoreSnapshot-1676973422348 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676973422348 completed! 2023-02-21 09:57:17,198 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 ) in 134 msec 2023-02-21 09:57:17,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-21 09:57:17,320 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676973422348, procId: 147 completed 2023-02-21 09:57:17,320 INFO [Listener at localhost/40243] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:17,321 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676973422348-1676973436779" 2023-02-21 09:57:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676973422348-1676973436779 2023-02-21 09:57:17,323 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,324 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-21 09:57:17,327 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676973422348 2023-02-21 09:57:17,327 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973437327"}]},"ts":"1676973437327"} 2023-02-21 09:57:17,328 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:17,331 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:17,332 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:17,333 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:17,333 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:17,333 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:17,334 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:17,334 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:17,334 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:17,334 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:17,334 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:17,335 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:17,335 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:17,335 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:17,335 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-21 09:57:17,484 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:17,484 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:17,484 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:17,485 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:17,485 DEBUG [jenkins-hbase4:41687] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:17,488 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:17,488 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:17,488 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973437488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973437488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973437488"}]},"ts":"1676973437488"} 2023-02-21 09:57:17,488 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:17,488 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:17,488 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973437488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973437488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973437488"}]},"ts":"1676973437488"} 2023-02-21 09:57:17,488 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973437488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973437488"}]},"ts":"1676973437488"} 2023-02-21 09:57:17,488 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:17,488 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973437488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973437488"}]},"ts":"1676973437488"} 2023-02-21 09:57:17,488 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437488"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973437488"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973437488"}]},"ts":"1676973437488"} 2023-02-21 09:57:17,489 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=149, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:17,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=154, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:17,490 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:17,491 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=153, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:17,493 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:17,493 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437493"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973437493"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973437493"}]},"ts":"1676973437493"} 2023-02-21 09:57:17,493 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=150, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:17,494 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:17,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-21 09:57:17,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:17,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:17,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:17,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,645 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:17,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,646 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,646 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:17,647 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:17,647 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,647 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:17,647 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:17,647 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:17,647 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:17,647 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:17,648 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:17,648 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:17,663 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:17,663 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:17,663 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:17,670 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:17,671 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:17,671 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 2023-02-21 09:57:17,671 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:17,671 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:17,671 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:17,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:17,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:17,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:17,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62950661, jitterRate=-0.061962053179740906}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:17,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:17,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72221346, jitterRate=0.07618191838264465}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:17,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68335180, jitterRate=0.018273532390594482}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:17,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:17,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:17,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=156, masterSystemTime=1676973437641 2023-02-21 09:57:17,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=155, masterSystemTime=1676973437641 2023-02-21 09:57:17,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=158, masterSystemTime=1676973437643 2023-02-21 09:57:17,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:17,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:17,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:17,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:17,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:17,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,681 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:17,681 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973437681"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973437681"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973437681"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973437681"}]},"ts":"1676973437681"} 2023-02-21 09:57:17,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:17,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:17,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:17,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:17,682 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:17,682 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973437681"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973437681"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973437681"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973437681"}]},"ts":"1676973437681"} 2023-02-21 09:57:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,682 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:17,682 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437682"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973437682"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973437682"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973437682"}]},"ts":"1676973437682"} 2023-02-21 09:57:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:17,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:17,683 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:17,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:17,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:17,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,685 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:17,685 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:17,685 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,686 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:17,687 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=154 2023-02-21 09:57:17,687 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=154, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 193 msec 2023-02-21 09:57:17,688 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=149 2023-02-21 09:57:17,688 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=149, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 195 msec 2023-02-21 09:57:17,689 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 356 msec 2023-02-21 09:57:17,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=153 2023-02-21 09:57:17,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=153, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539 in 194 msec 2023-02-21 09:57:17,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 357 msec 2023-02-21 09:57:17,691 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 359 msec 2023-02-21 09:57:17,692 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,692 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:17,692 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:17,693 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:17,693 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:17,693 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:17,693 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:17,700 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:17,703 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:17,712 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:17,716 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:17,716 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:17,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,718 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 2023-02-21 09:57:17,718 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:17,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,719 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:17,719 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:17,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:17,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68590380, jitterRate=0.022076308727264404}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:17,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:17,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:17,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:17,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=157, masterSystemTime=1676973437641 2023-02-21 09:57:17,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75434227, jitterRate=0.1240575760602951}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:17,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64340459, jitterRate=-0.04125244915485382}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:17,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:17,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:17,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=160, masterSystemTime=1676973437641 2023-02-21 09:57:17,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=159, masterSystemTime=1676973437643 2023-02-21 09:57:17,726 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:17,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:17,727 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:17,727 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437727"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973437727"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973437727"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973437727"}]},"ts":"1676973437727"} 2023-02-21 09:57:17,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:17,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:17,729 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:17,729 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437729"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973437729"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973437729"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973437729"}]},"ts":"1676973437729"} 2023-02-21 09:57:17,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:17,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:17,730 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:17,730 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973437730"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973437730"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973437730"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973437730"}]},"ts":"1676973437730"} 2023-02-21 09:57:17,731 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-21 09:57:17,731 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584 in 239 msec 2023-02-21 09:57:17,733 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-21 09:57:17,733 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614 in 237 msec 2023-02-21 09:57:17,733 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 400 msec 2023-02-21 09:57:17,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=150 2023-02-21 09:57:17,734 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 402 msec 2023-02-21 09:57:17,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=150, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 239 msec 2023-02-21 09:57:17,735 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-02-21 09:57:17,735 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 403 msec 2023-02-21 09:57:17,736 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973437736"}]},"ts":"1676973437736"} 2023-02-21 09:57:17,737 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:17,739 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676973422348' was successfully enabled. 2023-02-21 09:57:17,741 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676973422348 in 416 msec 2023-02-21 09:57:18,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-21 09:57:18,079 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 148 completed 2023-02-21 09:57:18,079 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(139): Connect 0x77b5b1f1 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:18,085 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1550b85e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:18,087 DEBUG [hconnection-0x2ed0e7fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:18,091 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38370, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:18,094 DEBUG [hconnection-0x2ed0e7fa-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:18,096 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39846, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:18,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,272 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,391 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b384ceb to 127.0.0.1:58856 2023-02-21 09:57:18,392 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:18,393 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c2934c9 to 127.0.0.1:58856 2023-02-21 09:57:18,393 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:18,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,482 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:18,485 DEBUG [hconnection-0x2ed0e7fa-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:18,487 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56168, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:18,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,721 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:18,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,964 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,160 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:19,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:19,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:21,217 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,218 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-21 09:57:21,222 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973441221"}]},"ts":"1676973441221"} 2023-02-21 09:57:21,223 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:21,225 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:21,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:21,230 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:21,230 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:21,230 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:21,230 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:21,230 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:21,230 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:21,230 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973441230"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973441230"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973441230"}]},"ts":"1676973441230"} 2023-02-21 09:57:21,231 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:21,231 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441231"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973441231"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973441231"}]},"ts":"1676973441231"} 2023-02-21 09:57:21,231 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:21,231 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:21,231 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441231"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973441231"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973441231"}]},"ts":"1676973441231"} 2023-02-21 09:57:21,231 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441231"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973441231"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973441231"}]},"ts":"1676973441231"} 2023-02-21 09:57:21,231 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:21,232 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973441231"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973441231"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973441231"}]},"ts":"1676973441231"} 2023-02-21 09:57:21,232 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=162, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:21,233 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=164, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:21,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=163, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:21,234 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:21,235 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=165, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:21,235 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:21,235 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441235"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973441235"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973441235"}]},"ts":"1676973441235"} 2023-02-21 09:57:21,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=167, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:21,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:21,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:21,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:21,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:21,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:21,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:21,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:21,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:21,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:21,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:21,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:21,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:21,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:21,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,397 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-21 09:57:21,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:21,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:21,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-21 09:57:21,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-21 09:57:21,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:21,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:21,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:21,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:21,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:21,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:21,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,400 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:21,401 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973441400"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973441400"}]},"ts":"1676973441400"} 2023-02-21 09:57:21,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:21,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:21,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:21,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:21,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,405 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:21,406 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973441402"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973441402"}]},"ts":"1676973441402"} 2023-02-21 09:57:21,406 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:21,406 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441406"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973441406"}]},"ts":"1676973441406"} 2023-02-21 09:57:21,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:21,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:21,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:21,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:21,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,411 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=162 2023-02-21 09:57:21,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=162, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,44835,1676973419584 in 173 msec 2023-02-21 09:57:21,413 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 184 msec 2023-02-21 09:57:21,415 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-21 09:57:21,415 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,46077,1676973419539 in 175 msec 2023-02-21 09:57:21,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=167 2023-02-21 09:57:21,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-21 09:57:21,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=167, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 175 msec 2023-02-21 09:57:21,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-21 09:57:21,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:21,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:21,419 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 187 msec 2023-02-21 09:57:21,419 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 188 msec 2023-02-21 09:57:21,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:21,420 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:21,420 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441420"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973441420"}]},"ts":"1676973441420"} 2023-02-21 09:57:21,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:21,420 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:21,421 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441420"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973441420"}]},"ts":"1676973441420"} 2023-02-21 09:57:21,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-21 09:57:21,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:21,423 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=165 2023-02-21 09:57:21,423 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=165, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584 in 186 msec 2023-02-21 09:57:21,424 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=164 2023-02-21 09:57:21,424 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=164, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,41011,1676973419614 in 189 msec 2023-02-21 09:57:21,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:21,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 195 msec 2023-02-21 09:57:21,424 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:21,424 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973441424"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973441424"}]},"ts":"1676973441424"} 2023-02-21 09:57:21,425 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 196 msec 2023-02-21 09:57:21,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=163 2023-02-21 09:57:21,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=163, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,46077,1676973419539 in 191 msec 2023-02-21 09:57:21,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=163, resume processing ppid=161 2023-02-21 09:57:21,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 198 msec 2023-02-21 09:57:21,429 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973441428"}]},"ts":"1676973441428"} 2023-02-21 09:57:21,430 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:21,432 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:21,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 215 msec 2023-02-21 09:57:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-21 09:57:21,473 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 161 completed 2023-02-21 09:57:21,477 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,486 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,487 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-21 09:57:21,494 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:21,494 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:21,494 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:21,494 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:21,494 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:21,494 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:21,497 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits] 2023-02-21 09:57:21,497 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits] 2023-02-21 09:57:21,497 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits] 2023-02-21 09:57:21,497 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits] 2023-02-21 09:57:21,497 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits] 2023-02-21 09:57:21,498 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits] 2023-02-21 09:57:21,501 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:21,501 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:21,501 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:21,502 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:21,503 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:21,503 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:21,504 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 2023-02-21 09:57:21,504 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:21,504 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 2023-02-21 09:57:21,505 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:21,505 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:21,506 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:21,509 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/27.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/27.seqid 2023-02-21 09:57:21,510 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/27.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/27.seqid 2023-02-21 09:57:21,510 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/27.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/27.seqid 2023-02-21 09:57:21,510 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/27.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/27.seqid 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/27.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/27.seqid 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/27.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/27.seqid 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:21,511 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:21,512 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:21,512 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676973422348 regions 2023-02-21 09:57:21,512 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650 2023-02-21 09:57:21,513 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf] 2023-02-21 09:57:21,515 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa 2023-02-21 09:57:21,516 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:21,517 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 2023-02-21 09:57:21,518 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:21,519 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c 2023-02-21 09:57:21,520 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:21,521 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:21,522 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb 2023-02-21 09:57:21,523 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e 2023-02-21 09:57:21,524 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:21,525 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 2023-02-21 09:57:21,526 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:21,526 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650 2023-02-21 09:57:21,528 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,549 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676973422348 from hbase:meta 2023-02-21 09:57:21,551 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676973422348' descriptor. 2023-02-21 09:57:21,551 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676973422348' from region states. 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973441552"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973441552"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973441552"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973441552"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973441552"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973441552"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,554 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-21 09:57:21,554 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}] 2023-02-21 09:57:21,555 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676973422348' as deleted. 2023-02-21 09:57:21,555 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676973441555"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:21,557 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676973422348 state from META 2023-02-21 09:57:21,559 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 in 81 msec 2023-02-21 09:57:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-21 09:57:21,740 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 174 completed 2023-02-21 09:57:21,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 ) 2023-02-21 09:57:21,768 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676973422348 as table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-21 09:57:21,781 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:21,783 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:21,783 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,784 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,786 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,789 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,791 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,793 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,794 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,796 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,798 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,800 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,801 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,803 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,804 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: eb310106c5f65634673574312633a7b9 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=a585cd7f02d60b76c310aa80c70f3a08 as a585cd7f02d60b76c310aa80c70f3a08 in snapshot snaptb1-1676973422348 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=dbcdd07ed31c76cf5ed03a249f4cf8d1 as dbcdd07ed31c76cf5ed03a249f4cf8d1 in snapshot snaptb1-1676973422348 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=74b647ed77adba33d8bf8ed03d9c05fc as 74b647ed77adba33d8bf8ed03d9c05fc in snapshot snaptb1-1676973422348 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=5863e48176a446e50692d5bcf8d380f1 as 5863e48176a446e50692d5bcf8d380f1 in snapshot snaptb1-1676973422348 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=5c80767ec66fcdbcf22028d0c0cfeee5 as 5c80767ec66fcdbcf22028d0c0cfeee5 in snapshot snaptb1-1676973422348 2023-02-21 09:57:21,806 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=eb310106c5f65634673574312633a7b9 as eb310106c5f65634673574312633a7b9 in snapshot snaptb1-1676973422348 2023-02-21 09:57:21,808 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:21,809 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:21,809 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:21,809 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:21,809 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:21,809 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676973422348', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:21,830 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:21,831 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2ffa7e7114804113a9806984d81b4e53 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,833 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:21,833 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:21,833 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 91b66786118a4ebf850b72bed26f0cae from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,833 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 1d0a8ea5761f44caa7dda024184b0758 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,834 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a912b8ec2a324a91924827dcf86ef9ec from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,836 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 5d93932811904d658444a38dedb905b4 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,840 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink b4cba6164e5646ebb1df5c1417884979 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,841 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:21,841 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,841 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,841 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:21,841 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,841 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:21,841 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:21,842 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:21,842 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 45fb0f2a211b4901a56150a2b662707f from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,842 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:21,842 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,842 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,842 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:21,842 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,842 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:21,842 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:21,844 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:21,844 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2d39b975032e413ab7e72e6533951b3e from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,844 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:21,844 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,844 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,844 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:21,844 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,844 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:21,844 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:21,845 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d65bd76661554d98a17951ff943be720 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,846 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a52c1f52ad274962a0a35c91207435f4 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:21,847 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:21,847 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,847 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,847 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:21,847 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,847 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:21,847 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:21,848 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:21,849 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,849 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,849 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:21,849 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,849 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:21,849 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-21 09:57:22,238 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,238 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 03e1eeaaf8564145a9dc0fa8a88a2d59 from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:22,241 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eaff9d6ee57e4a58a133e8233854e20f from cloned region in snapshot snaptb1-1676973422348 to table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:22,244 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:22,244 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,244 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,244 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:22,244 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,244 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,244 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:22,244 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 2023-02-21 09:57:22,244 INFO [PEWorker-3] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676973422348 on table=testRestoreSnapshot-1676973422348 completed! 2023-02-21 09:57:22,249 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973442249"}]},"ts":"1676973442249"} 2023-02-21 09:57:22,249 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973442249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973442249"}]},"ts":"1676973442249"} 2023-02-21 09:57:22,249 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973442249"}]},"ts":"1676973442249"} 2023-02-21 09:57:22,249 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973442249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973442249"}]},"ts":"1676973442249"} 2023-02-21 09:57:22,249 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973442249"}]},"ts":"1676973442249"} 2023-02-21 09:57:22,249 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442249"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973442249"}]},"ts":"1676973442249"} 2023-02-21 09:57:22,251 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-21 09:57:22,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973442252"}]},"ts":"1676973442252"} 2023-02-21 09:57:22,253 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLING in hbase:meta 2023-02-21 09:57:22,257 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:22,258 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:22,258 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:22,258 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:22,258 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:22,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN}] 2023-02-21 09:57:22,260 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN 2023-02-21 09:57:22,260 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN 2023-02-21 09:57:22,260 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN 2023-02-21 09:57:22,260 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN 2023-02-21 09:57:22,261 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=false 2023-02-21 09:57:22,261 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:22,261 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=false 2023-02-21 09:57:22,261 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=false 2023-02-21 09:57:22,262 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN 2023-02-21 09:57:22,262 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN 2023-02-21 09:57:22,263 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=false 2023-02-21 09:57:22,263 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:22,411 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:22,414 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:22,414 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:22,414 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973442414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973442414"}]},"ts":"1676973442414"} 2023-02-21 09:57:22,414 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:22,414 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:22,415 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973442414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973442414"}]},"ts":"1676973442414"} 2023-02-21 09:57:22,414 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:22,415 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973442414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973442414"}]},"ts":"1676973442414"} 2023-02-21 09:57:22,415 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973442414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973442414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973442414"}]},"ts":"1676973442414"} 2023-02-21 09:57:22,415 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973442414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973442414"}]},"ts":"1676973442414"} 2023-02-21 09:57:22,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=178, state=RUNNABLE; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:22,418 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=179, state=RUNNABLE; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:22,419 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=180, state=RUNNABLE; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:22,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=181, state=RUNNABLE; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:22,420 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=177, state=RUNNABLE; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:22,421 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:22,421 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973442421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973442421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973442421"}]},"ts":"1676973442421"} 2023-02-21 09:57:22,422 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=176, state=RUNNABLE; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:22,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-21 09:57:22,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:22,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:22,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:22,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:22,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:22,574 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,574 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,574 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:22,574 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf 2023-02-21 09:57:22,574 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:22,574 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf 2023-02-21 09:57:22,575 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,575 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5863e48176a446e50692d5bcf8d380f1 columnFamilyName cf 2023-02-21 09:57:22,575 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74b647ed77adba33d8bf8ed03d9c05fc columnFamilyName cf 2023-02-21 09:57:22,575 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:22,575 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf 2023-02-21 09:57:22,576 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c80767ec66fcdbcf22028d0c0cfeee5 columnFamilyName cf 2023-02-21 09:57:22,584 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:22,584 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:22,584 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:22,590 DEBUG [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 2023-02-21 09:57:22,590 INFO [StoreOpener-5863e48176a446e50692d5bcf8d380f1-1] regionserver.HStore(310): Store=5863e48176a446e50692d5bcf8d380f1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:22,590 DEBUG [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 2023-02-21 09:57:22,591 INFO [StoreOpener-74b647ed77adba33d8bf8ed03d9c05fc-1] regionserver.HStore(310): Store=74b647ed77adba33d8bf8ed03d9c05fc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:22,591 DEBUG [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:22,591 INFO [StoreOpener-5c80767ec66fcdbcf22028d0c0cfeee5-1] regionserver.HStore(310): Store=5c80767ec66fcdbcf22028d0c0cfeee5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:22,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:22,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:22,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-21 09:57:22,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-21 09:57:22,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-21 09:57:22,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5863e48176a446e50692d5bcf8d380f1; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60199076, jitterRate=-0.10296386480331421}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:22,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:22,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 74b647ed77adba33d8bf8ed03d9c05fc; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62890271, jitterRate=-0.06286193430423737}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:22,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:22,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5c80767ec66fcdbcf22028d0c0cfeee5; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64592736, jitterRate=-0.037493228912353516}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:22,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:22,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1., pid=185, masterSystemTime=1676973442569 2023-02-21 09:57:22,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5., pid=184, masterSystemTime=1676973442572 2023-02-21 09:57:22,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc., pid=183, masterSystemTime=1676973442570 2023-02-21 09:57:22,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:22,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:22,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:22,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:22,599 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:22,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,599 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973442599"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973442599"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973442599"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973442599"}]},"ts":"1676973442599"} 2023-02-21 09:57:22,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:22,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:22,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:22,600 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,600 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442600"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973442600"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973442600"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973442600"}]},"ts":"1676973442600"} 2023-02-21 09:57:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:22,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:22,600 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:22,601 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:22,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676973422348 dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,601 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442601"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973442601"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973442601"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973442601"}]},"ts":"1676973442601"} 2023-02-21 09:57:22,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:22,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,602 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:22,602 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf 2023-02-21 09:57:22,602 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,602 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a585cd7f02d60b76c310aa80c70f3a08 columnFamilyName cf 2023-02-21 09:57:22,603 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,603 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:22,603 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf 2023-02-21 09:57:22,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=181 2023-02-21 09:57:22,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=181, state=SUCCESS; OpenRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,46077,1676973419539 in 180 msec 2023-02-21 09:57:22,604 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb310106c5f65634673574312633a7b9 columnFamilyName cf 2023-02-21 09:57:22,606 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=179 2023-02-21 09:57:22,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=179, state=SUCCESS; OpenRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584 in 185 msec 2023-02-21 09:57:22,607 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:22,607 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf 2023-02-21 09:57:22,607 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, ASSIGN in 345 msec 2023-02-21 09:57:22,608 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dbcdd07ed31c76cf5ed03a249f4cf8d1 columnFamilyName cf 2023-02-21 09:57:22,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=180 2023-02-21 09:57:22,609 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, ASSIGN in 349 msec 2023-02-21 09:57:22,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=180, state=SUCCESS; OpenRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 184 msec 2023-02-21 09:57:22,610 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, ASSIGN in 351 msec 2023-02-21 09:57:22,616 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:22,616 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:22,618 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:22,619 DEBUG [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:22,620 INFO [StoreOpener-eb310106c5f65634673574312633a7b9-1] regionserver.HStore(310): Store=eb310106c5f65634673574312633a7b9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:22,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,622 DEBUG [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:22,622 INFO [StoreOpener-a585cd7f02d60b76c310aa80c70f3a08-1] regionserver.HStore(310): Store=a585cd7f02d60b76c310aa80c70f3a08/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:22,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,624 DEBUG [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:22,624 INFO [StoreOpener-dbcdd07ed31c76cf5ed03a249f4cf8d1-1] regionserver.HStore(310): Store=dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:22,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb310106c5f65634673574312633a7b9 2023-02-21 09:57:22,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:22,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:22,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-21 09:57:22,627 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb310106c5f65634673574312633a7b9; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59108944, jitterRate=-0.11920809745788574}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:22,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:22,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-21 09:57:22,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a585cd7f02d60b76c310aa80c70f3a08; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74985557, jitterRate=0.11737187206745148}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:22,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9., pid=186, masterSystemTime=1676973442570 2023-02-21 09:57:22,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:22,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08., pid=182, masterSystemTime=1676973442569 2023-02-21 09:57:22,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-21 09:57:22,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:22,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:22,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dbcdd07ed31c76cf5ed03a249f4cf8d1; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73599070, jitterRate=0.09671160578727722}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:22,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:22,630 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:22,630 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442630"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973442630"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973442630"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973442630"}]},"ts":"1676973442630"} 2023-02-21 09:57:22,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:22,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:22,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1., pid=187, masterSystemTime=1676973442572 2023-02-21 09:57:22,632 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:22,632 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973442632"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973442632"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973442632"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973442632"}]},"ts":"1676973442632"} 2023-02-21 09:57:22,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:22,634 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:22,634 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973442634"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973442634"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973442634"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973442634"}]},"ts":"1676973442634"} 2023-02-21 09:57:22,635 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=177 2023-02-21 09:57:22,635 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=177, state=SUCCESS; OpenRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,44835,1676973419584 in 212 msec 2023-02-21 09:57:22,636 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=178 2023-02-21 09:57:22,636 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=178, state=SUCCESS; OpenRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,46077,1676973419539 in 218 msec 2023-02-21 09:57:22,637 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, ASSIGN in 377 msec 2023-02-21 09:57:22,637 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=176 2023-02-21 09:57:22,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, ASSIGN in 378 msec 2023-02-21 09:57:22,637 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=176, state=SUCCESS; OpenRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 214 msec 2023-02-21 09:57:22,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=176, resume processing ppid=175 2023-02-21 09:57:22,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, ASSIGN in 379 msec 2023-02-21 09:57:22,640 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973442639"}]},"ts":"1676973442639"} 2023-02-21 09:57:22,641 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=ENABLED in hbase:meta 2023-02-21 09:57:22,646 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676973422348 snapshot=name: "snaptb1-1676973422348" table: "testRestoreSnapshot-1676973422348" creation_time: 1676973431373 type: DISABLED version: 2 ttl: 0 ) in 883 msec 2023-02-21 09:57:23,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-21 09:57:23,274 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676973422348, procId: 175 completed 2023-02-21 09:57:23,275 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(139): Connect 0x4672389c to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:23,283 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ff51970, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:23,285 DEBUG [hconnection-0x4c9ce563-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:23,286 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38378, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:23,289 DEBUG [hconnection-0x4c9ce563-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:23,290 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:23,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,453 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:23,456 DEBUG [hconnection-0x4c9ce563-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:23,457 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35016, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:23,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,604 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:23,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,789 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:23,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,983 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:23,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,143 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,815 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77b5b1f1 to 127.0.0.1:58856 2023-02-21 09:57:24,816 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:24,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:25,943 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676973422348 2023-02-21 09:57:25,944 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676973422348 2023-02-21 09:57:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-21 09:57:25,947 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973445947"}]},"ts":"1676973445947"} 2023-02-21 09:57:25,949 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLING in hbase:meta 2023-02-21 09:57:25,951 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676973422348 to state=DISABLING 2023-02-21 09:57:25,951 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN}] 2023-02-21 09:57:25,953 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN 2023-02-21 09:57:25,953 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN 2023-02-21 09:57:25,954 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN 2023-02-21 09:57:25,954 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN 2023-02-21 09:57:25,954 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN 2023-02-21 09:57:25,954 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:25,954 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973445954"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973445954"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973445954"}]},"ts":"1676973445954"} 2023-02-21 09:57:25,955 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:25,955 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:25,955 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973445955"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973445955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973445955"}]},"ts":"1676973445955"} 2023-02-21 09:57:25,955 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973445955"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973445955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973445955"}]},"ts":"1676973445955"} 2023-02-21 09:57:25,955 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:25,955 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973445955"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973445955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973445955"}]},"ts":"1676973445955"} 2023-02-21 09:57:25,955 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:25,955 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973445955"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973445955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973445955"}]},"ts":"1676973445955"} 2023-02-21 09:57:25,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=190, state=RUNNABLE; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:25,957 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=191, state=RUNNABLE; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:25,958 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=189, state=RUNNABLE; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:25,960 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=192, state=RUNNABLE; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:25,961 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN 2023-02-21 09:57:25,961 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=194, state=RUNNABLE; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:25,962 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:25,962 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973445962"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973445962"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973445962"}]},"ts":"1676973445962"} 2023-02-21 09:57:25,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:26,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb310106c5f65634673574312633a7b9 2023-02-21 09:57:26,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:26,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb310106c5f65634673574312633a7b9, disabling compactions & flushes 2023-02-21 09:57:26,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:26,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:26,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a585cd7f02d60b76c310aa80c70f3a08, disabling compactions & flushes 2023-02-21 09:57:26,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. after waiting 0 ms 2023-02-21 09:57:26,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:26,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:26,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:26,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. after waiting 0 ms 2023-02-21 09:57:26,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:26,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:26,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5c80767ec66fcdbcf22028d0c0cfeee5, disabling compactions & flushes 2023-02-21 09:57:26,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:26,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:26,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. after waiting 0 ms 2023-02-21 09:57:26,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:26,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-21 09:57:26,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-21 09:57:26,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9. 2023-02-21 09:57:26,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb310106c5f65634673574312633a7b9: 2023-02-21 09:57:26,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08. 2023-02-21 09:57:26,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a585cd7f02d60b76c310aa80c70f3a08: 2023-02-21 09:57:26,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb310106c5f65634673574312633a7b9 2023-02-21 09:57:26,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:26,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-21 09:57:26,125 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=eb310106c5f65634673574312633a7b9, regionState=CLOSED 2023-02-21 09:57:26,125 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973446125"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446125"}]},"ts":"1676973446125"} 2023-02-21 09:57:26,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:26,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:26,126 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=a585cd7f02d60b76c310aa80c70f3a08, regionState=CLOSED 2023-02-21 09:57:26,126 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973446126"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446126"}]},"ts":"1676973446126"} 2023-02-21 09:57:26,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 74b647ed77adba33d8bf8ed03d9c05fc, disabling compactions & flushes 2023-02-21 09:57:26,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5863e48176a446e50692d5bcf8d380f1, disabling compactions & flushes 2023-02-21 09:57:26,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:26,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:26,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. after waiting 0 ms 2023-02-21 09:57:26,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:26,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:26,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:26,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. after waiting 0 ms 2023-02-21 09:57:26,128 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:26,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5. 2023-02-21 09:57:26,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5c80767ec66fcdbcf22028d0c0cfeee5: 2023-02-21 09:57:26,135 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=190 2023-02-21 09:57:26,135 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=190, state=SUCCESS; CloseRegionProcedure eb310106c5f65634673574312633a7b9, server=jenkins-hbase4.apache.org,44835,1676973419584 in 170 msec 2023-02-21 09:57:26,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:26,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:26,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dbcdd07ed31c76cf5ed03a249f4cf8d1, disabling compactions & flushes 2023-02-21 09:57:26,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:26,137 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=191 2023-02-21 09:57:26,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:26,137 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=191, state=SUCCESS; CloseRegionProcedure a585cd7f02d60b76c310aa80c70f3a08, server=jenkins-hbase4.apache.org,46077,1676973419539 in 171 msec 2023-02-21 09:57:26,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. after waiting 0 ms 2023-02-21 09:57:26,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:26,137 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=5c80767ec66fcdbcf22028d0c0cfeee5, regionState=CLOSED 2023-02-21 09:57:26,138 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973446137"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446137"}]},"ts":"1676973446137"} 2023-02-21 09:57:26,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=a585cd7f02d60b76c310aa80c70f3a08, UNASSIGN in 186 msec 2023-02-21 09:57:26,138 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=eb310106c5f65634673574312633a7b9, UNASSIGN in 184 msec 2023-02-21 09:57:26,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-21 09:57:26,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-21 09:57:26,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc. 2023-02-21 09:57:26,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 74b647ed77adba33d8bf8ed03d9c05fc: 2023-02-21 09:57:26,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1. 2023-02-21 09:57:26,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5863e48176a446e50692d5bcf8d380f1: 2023-02-21 09:57:26,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:26,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-21 09:57:26,141 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=74b647ed77adba33d8bf8ed03d9c05fc, regionState=CLOSED 2023-02-21 09:57:26,141 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure 5c80767ec66fcdbcf22028d0c0cfeee5, server=jenkins-hbase4.apache.org,41011,1676973419614 in 176 msec 2023-02-21 09:57:26,141 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676973446141"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446141"}]},"ts":"1676973446141"} 2023-02-21 09:57:26,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:26,142 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=5863e48176a446e50692d5bcf8d380f1, regionState=CLOSED 2023-02-21 09:57:26,142 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973446142"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446142"}]},"ts":"1676973446142"} 2023-02-21 09:57:26,143 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5c80767ec66fcdbcf22028d0c0cfeee5, UNASSIGN in 190 msec 2023-02-21 09:57:26,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-21 09:57:26,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1. 2023-02-21 09:57:26,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dbcdd07ed31c76cf5ed03a249f4cf8d1: 2023-02-21 09:57:26,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=192 2023-02-21 09:57:26,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=192, state=SUCCESS; CloseRegionProcedure 74b647ed77adba33d8bf8ed03d9c05fc, server=jenkins-hbase4.apache.org,44835,1676973419584 in 185 msec 2023-02-21 09:57:26,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:26,145 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=194 2023-02-21 09:57:26,145 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=194, state=SUCCESS; CloseRegionProcedure 5863e48176a446e50692d5bcf8d380f1, server=jenkins-hbase4.apache.org,46077,1676973419539 in 183 msec 2023-02-21 09:57:26,146 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=dbcdd07ed31c76cf5ed03a249f4cf8d1, regionState=CLOSED 2023-02-21 09:57:26,146 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=74b647ed77adba33d8bf8ed03d9c05fc, UNASSIGN in 194 msec 2023-02-21 09:57:26,146 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676973446146"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446146"}]},"ts":"1676973446146"} 2023-02-21 09:57:26,146 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=5863e48176a446e50692d5bcf8d380f1, UNASSIGN in 194 msec 2023-02-21 09:57:26,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=189 2023-02-21 09:57:26,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=189, state=SUCCESS; CloseRegionProcedure dbcdd07ed31c76cf5ed03a249f4cf8d1, server=jenkins-hbase4.apache.org,41011,1676973419614 in 189 msec 2023-02-21 09:57:26,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=189, resume processing ppid=188 2023-02-21 09:57:26,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676973422348, region=dbcdd07ed31c76cf5ed03a249f4cf8d1, UNASSIGN in 197 msec 2023-02-21 09:57:26,150 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973446149"}]},"ts":"1676973446149"} 2023-02-21 09:57:26,150 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676973422348, state=DISABLED in hbase:meta 2023-02-21 09:57:26,152 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676973422348 to state=DISABLED 2023-02-21 09:57:26,153 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676973422348 in 208 msec 2023-02-21 09:57:26,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-21 09:57:26,198 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 188 completed 2023-02-21 09:57:26,199 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,201 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,202 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-21 09:57:26,208 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:26,208 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:26,208 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:26,208 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:26,208 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:26,208 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:26,209 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits] 2023-02-21 09:57:26,210 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits] 2023-02-21 09:57:26,210 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits] 2023-02-21 09:57:26,210 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits] 2023-02-21 09:57:26,211 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits] 2023-02-21 09:57:26,211 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits] 2023-02-21 09:57:26,214 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441501 currentMtime: 1676973441836 2023-02-21 09:57:26,214 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441501 currentMtime: 1676973442241 2023-02-21 09:57:26,214 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441500 currentMtime: 1676973441834 2023-02-21 09:57:26,215 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441499 currentMtime: 1676973441844 2023-02-21 09:57:26,215 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae.1676973446210. 2023-02-21 09:57:26,215 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53.1676973446210. 2023-02-21 09:57:26,215 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441500 currentMtime: 1676973441836 2023-02-21 09:57:26,215 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59.1676973446209. 2023-02-21 09:57:26,215 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441500 currentMtime: 1676973441846 2023-02-21 09:57:26,215 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f.1676973446210. 2023-02-21 09:57:26,215 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758.1676973446211. 2023-02-21 09:57:26,216 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e.1676973446211. 2023-02-21 09:57:26,216 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-91b66786118a4ebf850b72bed26f0cae 2023-02-21 09:57:26,216 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-03e1eeaaf8564145a9dc0fa8a88a2d59 2023-02-21 09:57:26,216 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-2ffa7e7114804113a9806984d81b4e53 2023-02-21 09:57:26,216 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-1d0a8ea5761f44caa7dda024184b0758 2023-02-21 09:57:26,216 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-45fb0f2a211b4901a56150a2b662707f 2023-02-21 09:57:26,217 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-2d39b975032e413ab7e72e6533951b3e 2023-02-21 09:57:26,218 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441503 currentMtime: 1676973441840 2023-02-21 09:57:26,218 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441504 currentMtime: 1676973441844 2023-02-21 09:57:26,218 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441502 currentMtime: 1676973441846 2023-02-21 09:57:26,218 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441505 currentMtime: 1676973442243 2023-02-21 09:57:26,219 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec.1676973446210. 2023-02-21 09:57:26,219 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979.1676973446210. 2023-02-21 09:57:26,219 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720.1676973446210. 2023-02-21 09:57:26,219 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441502 currentMtime: 1676973441848 2023-02-21 09:57:26,219 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441503 currentMtime: 1676973441841 2023-02-21 09:57:26,219 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f.1676973446209. 2023-02-21 09:57:26,219 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/cf/testRestoreSnapshot-1676973422348=a585cd7f02d60b76c310aa80c70f3a08-a912b8ec2a324a91924827dcf86ef9ec 2023-02-21 09:57:26,219 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4.1676973446211. 2023-02-21 09:57:26,220 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4.1676973446211. 2023-02-21 09:57:26,220 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/cf/testRestoreSnapshot-1676973422348=74b647ed77adba33d8bf8ed03d9c05fc-d65bd76661554d98a17951ff943be720 2023-02-21 09:57:26,220 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/cf/testRestoreSnapshot-1676973422348=eb310106c5f65634673574312633a7b9-b4cba6164e5646ebb1df5c1417884979 2023-02-21 09:57:26,220 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/cf/testRestoreSnapshot-1676973422348=dbcdd07ed31c76cf5ed03a249f4cf8d1-eaff9d6ee57e4a58a133e8233854e20f 2023-02-21 09:57:26,221 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/cf/testRestoreSnapshot-1676973422348=5c80767ec66fcdbcf22028d0c0cfeee5-a52c1f52ad274962a0a35c91207435f4 2023-02-21 09:57:26,221 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/cf/testRestoreSnapshot-1676973422348=5863e48176a446e50692d5bcf8d380f1-5d93932811904d658444a38dedb905b4 2023-02-21 09:57:26,222 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08/recovered.edits/18.seqid 2023-02-21 09:57:26,223 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc/recovered.edits/18.seqid 2023-02-21 09:57:26,223 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/a585cd7f02d60b76c310aa80c70f3a08 2023-02-21 09:57:26,223 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/74b647ed77adba33d8bf8ed03d9c05fc 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5/recovered.edits/18.seqid 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9/recovered.edits/18.seqid 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1/recovered.edits/18.seqid 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5c80767ec66fcdbcf22028d0c0cfeee5 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1/recovered.edits/18.seqid 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/eb310106c5f65634673574312633a7b9 2023-02-21 09:57:26,224 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/dbcdd07ed31c76cf5ed03a249f4cf8d1 2023-02-21 09:57:26,225 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testRestoreSnapshot-1676973422348/5863e48176a446e50692d5bcf8d380f1 2023-02-21 09:57:26,225 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676973422348 regions 2023-02-21 09:57:26,225 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650 2023-02-21 09:57:26,226 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf] 2023-02-21 09:57:26,228 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441514 currentMtime: 1676973441786 2023-02-21 09:57:26,228 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa.1676973446226. 2023-02-21 09:57:26,229 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302212fb61906b5434c1495d39dccc7b285aa 2023-02-21 09:57:26,230 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441515 currentMtime: 1676973441788 2023-02-21 09:57:26,231 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167.1676973446226. 2023-02-21 09:57:26,232 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-a87ff679a2f3e71d9181a67b7542122c202302214b854b85528845d393495778ba4d6167 2023-02-21 09:57:26,236 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441516 currentMtime: 1676973441791 2023-02-21 09:57:26,237 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6.1676973446226. 2023-02-21 09:57:26,238 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221153859f1bbac4bcba87774c4da9a4ac6 2023-02-21 09:57:26,239 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441517 currentMtime: 1676973441792 2023-02-21 09:57:26,239 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8.1676973446226. 2023-02-21 09:57:26,240 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c4ca4238a0b923820dcc509a6f75849b20230221753acdfba0e5439f9878b2a80c67fee8 2023-02-21 09:57:26,241 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441518 currentMtime: 1676973441794 2023-02-21 09:57:26,242 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c.1676973446226. 2023-02-21 09:57:26,242 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c20230221565c01178be44ebab26d1ab183aa5e5c 2023-02-21 09:57:26,243 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441519 currentMtime: 1676973441796 2023-02-21 09:57:26,244 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a.1676973446226. 2023-02-21 09:57:26,245 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-c81e728d9d4c2f636f067f89cc14862c202302219144698b87664160adeeb8afb505be4a 2023-02-21 09:57:26,246 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441520 currentMtime: 1676973441798 2023-02-21 09:57:26,246 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30.1676973446226. 2023-02-21 09:57:26,247 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e2023022119ebf99f6e97413bbd1e55aca04a3e30 2023-02-21 09:57:26,248 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441521 currentMtime: 1676973441799 2023-02-21 09:57:26,249 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb.1676973446226. 2023-02-21 09:57:26,249 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-d41d8cd98f00b204e9800998ecf8427e20230221c1105c540a1447fc8a9cf1bf20adbdbb 2023-02-21 09:57:26,251 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441522 currentMtime: 1676973441801 2023-02-21 09:57:26,251 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e.1676973446226. 2023-02-21 09:57:26,252 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221574680d4b58f45e69cb26c372c0ec57e 2023-02-21 09:57:26,253 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441523 currentMtime: 1676973441803 2023-02-21 09:57:26,254 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb.1676973446226. 2023-02-21 09:57:26,254 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-e4da3b7fbbce2345d7772b0674a318d520230221dba0c1bd4953416492fd42071ed37bdb 2023-02-21 09:57:26,255 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441524 currentMtime: 1676973441804 2023-02-21 09:57:26,256 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38.1676973446226. 2023-02-21 09:57:26,257 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf3202302213946fca06a5649478d0ee876c715aa38 2023-02-21 09:57:26,257 ERROR [PEWorker-4] backup.HFileArchiver(523): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388. archiveLen: 0 currentLen: 0 archiveMtime: 1676973441525 currentMtime: 1676973441806 2023-02-21 09:57:26,258 INFO [PEWorker-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388.1676973446226. 2023-02-21 09:57:26,259 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650/cf/testRestoreSnapshot-1676973422348=9cd55b007e40a5b23f7970090b429650-eccbc87e4b5ce2fe28308fd9f2a7baf320230221bf2686457b034afa99472991228ff388 2023-02-21 09:57:26,259 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testRestoreSnapshot-1676973422348/9cd55b007e40a5b23f7970090b429650 2023-02-21 09:57:26,261 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,264 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676973422348 from hbase:meta 2023-02-21 09:57:26,270 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676973422348' descriptor. 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676973422348' from region states. 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973446271"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973446271"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973446271"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973446271"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973446271"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,271 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973446271"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,273 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-21 09:57:26,273 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => dbcdd07ed31c76cf5ed03a249f4cf8d1, NAME => 'testRestoreSnapshot-1676973422348,,1676973422371.dbcdd07ed31c76cf5ed03a249f4cf8d1.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => eb310106c5f65634673574312633a7b9, NAME => 'testRestoreSnapshot-1676973422348,1,1676973422371.eb310106c5f65634673574312633a7b9.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => a585cd7f02d60b76c310aa80c70f3a08, NAME => 'testRestoreSnapshot-1676973422348,2,1676973422371.a585cd7f02d60b76c310aa80c70f3a08.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 74b647ed77adba33d8bf8ed03d9c05fc, NAME => 'testRestoreSnapshot-1676973422348,3,1676973422371.74b647ed77adba33d8bf8ed03d9c05fc.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 5c80767ec66fcdbcf22028d0c0cfeee5, NAME => 'testRestoreSnapshot-1676973422348,4,1676973422371.5c80767ec66fcdbcf22028d0c0cfeee5.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 5863e48176a446e50692d5bcf8d380f1, NAME => 'testRestoreSnapshot-1676973422348,5,1676973422371.5863e48176a446e50692d5bcf8d380f1.', STARTKEY => '5', ENDKEY => ''}] 2023-02-21 09:57:26,273 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676973422348' as deleted. 2023-02-21 09:57:26,273 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676973422348","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676973446273"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:26,274 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676973422348 state from META 2023-02-21 09:57:26,277 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 2023-02-21 09:57:26,278 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676973422348 in 78 msec 2023-02-21 09:57:26,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-21 09:57:26,453 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676973422348, procId: 201 completed 2023-02-21 09:57:26,458 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676973422348" 2023-02-21 09:57:26,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676973422348 2023-02-21 09:57:26,461 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676973422348" 2023-02-21 09:57:26,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676973422348 2023-02-21 09:57:26,463 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676973422348" 2023-02-21 09:57:26,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676973422348 2023-02-21 09:57:26,492 INFO [Listener at localhost/40243] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=424 (was 360) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_59681323_15 at /127.0.0.1:56010 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1895860658_15 at /127.0.0.1:41224 [Waiting for operation #204] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_927675210_15 at /127.0.0.1:59990 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_59681323_15 at /127.0.0.1:34982 [Waiting for operation #486] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58856@0x4672389c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/609283332.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_59681323_15 at /127.0.0.1:60048 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x72233ac4-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_927675210_15 at /127.0.0.1:34974 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58856@0x4672389c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x72233ac4-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7b9b1635-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1895860658_15 at /127.0.0.1:34976 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1895860658_15 at /127.0.0.1:41302 [Waiting for operation #423] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1565837293_15 at /127.0.0.1:40978 [Waiting for operation #719] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_59681323_15 at /127.0.0.1:60058 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_927675210_15 at /127.0.0.1:41288 [Waiting for operation #421] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1565837293_15 at /127.0.0.1:59984 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_59681323_15 at /127.0.0.1:60072 [Waiting for operation #496] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1565837293_15 at /127.0.0.1:34988 [Waiting for operation #528] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:58856@0x4672389c-SendThread(127.0.0.1:58856) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_59681323_15 at /127.0.0.1:56090 [Waiting for operation #525] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4c9ce563-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x72233ac4-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1895860658_15 at /127.0.0.1:56082 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x72233ac4-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=814 (was 697) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=249 (was 246) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 174), AvailableMemoryMB=5230 (was 5652) 2023-02-21 09:57:26,506 INFO [Listener at localhost/40243] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=424, OpenFileDescriptor=814, MaxFileDescriptor=60000, SystemLoadAverage=249, ProcessCount=172, AvailableMemoryMB=5229 2023-02-21 09:57:26,509 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-21 09:57:26,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:26,511 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-21 09:57:26,511 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676973446507" procId is: 202 2023-02-21 09:57:26,512 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-21 09:57:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-21 09:57:26,515 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:26,515 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:26,515 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:26,515 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:26,515 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:26,515 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:26,516 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb empty. 2023-02-21 09:57:26,516 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 empty. 2023-02-21 09:57:26,516 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a empty. 2023-02-21 09:57:26,516 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 empty. 2023-02-21 09:57:26,516 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab empty. 2023-02-21 09:57:26,516 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e empty. 2023-02-21 09:57:26,517 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:26,517 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:26,517 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:26,517 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:26,517 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:26,517 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:26,517 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676973446507 regions 2023-02-21 09:57:26,526 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/.tabledesc/.tableinfo.0000000001 2023-02-21 09:57:26,528 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(7671): creating {ENCODED => b51ce6c2c565e8d6b692d66c994428bb, NAME => 'testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:26,528 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(7671): creating {ENCODED => 9bdca281384ff4be099bfeae0c42d2c6, NAME => 'testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:26,529 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(7671): creating {ENCODED => 14df7364e6a8c8dc1107428e33ce230a, NAME => 'testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:26,552 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:26,552 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1603): Closing 9bdca281384ff4be099bfeae0c42d2c6, disabling compactions & flushes 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1603): Closing b51ce6c2c565e8d6b692d66c994428bb, disabling compactions & flushes 2023-02-21 09:57:26,553 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:26,553 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. after waiting 0 ms 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:26,553 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. after waiting 0 ms 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1557): Region close journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:26,553 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:26,553 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1557): Region close journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:26,554 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(7671): creating {ENCODED => ca9f39730d0c524ed83cab2e18c641c8, NAME => 'testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:26,554 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(7671): creating {ENCODED => 6472b4dfac0a398ae332dd5f9936afab, NAME => 'testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:26,561 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:26,561 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1603): Closing ca9f39730d0c524ed83cab2e18c641c8, disabling compactions & flushes 2023-02-21 09:57:26,561 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:26,561 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:26,561 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. after waiting 0 ms 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:26,562 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1557): Region close journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1603): Closing 6472b4dfac0a398ae332dd5f9936afab, disabling compactions & flushes 2023-02-21 09:57:26,562 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(7671): creating {ENCODED => 81b4d0d7d383dbd3ed3b6da2e1fab12e, NAME => 'testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676973446507', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp 2023-02-21 09:57:26,562 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. after waiting 0 ms 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:26,562 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:26,562 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-0] regionserver.HRegion(1557): Region close journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:26,568 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:26,568 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1603): Closing 81b4d0d7d383dbd3ed3b6da2e1fab12e, disabling compactions & flushes 2023-02-21 09:57:26,569 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:26,569 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:26,569 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. after waiting 0 ms 2023-02-21 09:57:26,569 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:26,569 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:26,569 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-2] regionserver.HRegion(1557): Region close journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-21 09:57:26,900 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-21 09:57:26,953 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:26,953 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1603): Closing 14df7364e6a8c8dc1107428e33ce230a, disabling compactions & flushes 2023-02-21 09:57:26,953 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:26,954 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:26,954 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. after waiting 0 ms 2023-02-21 09:57:26,954 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:26,954 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:26,954 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676973446507-pool-1] regionserver.HRegion(1557): Region close journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:26,956 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 execute state=CREATE_TABLE_ADD_TO_META 2023-02-21 09:57:26,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973446957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446957"}]},"ts":"1676973446957"} 2023-02-21 09:57:26,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973446957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446957"}]},"ts":"1676973446957"} 2023-02-21 09:57:26,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973446957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446957"}]},"ts":"1676973446957"} 2023-02-21 09:57:26,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973446957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446957"}]},"ts":"1676973446957"} 2023-02-21 09:57:26,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973446957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446957"}]},"ts":"1676973446957"} 2023-02-21 09:57:26,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973446957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973446957"}]},"ts":"1676973446957"} 2023-02-21 09:57:26,961 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-21 09:57:26,962 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-21 09:57:26,962 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973446962"}]},"ts":"1676973446962"} 2023-02-21 09:57:26,963 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=ENABLING in hbase:meta 2023-02-21 09:57:26,967 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-21 09:57:26,969 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 09:57:26,969 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-21 09:57:26,969 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-21 09:57:26,969 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-21 09:57:26,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN}] 2023-02-21 09:57:26,971 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN 2023-02-21 09:57:26,972 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN 2023-02-21 09:57:26,972 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN 2023-02-21 09:57:26,972 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN 2023-02-21 09:57:26,972 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN 2023-02-21 09:57:26,973 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=false 2023-02-21 09:57:26,973 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=false 2023-02-21 09:57:26,973 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:26,973 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=false 2023-02-21 09:57:26,973 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=false 2023-02-21 09:57:26,973 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN 2023-02-21 09:57:26,974 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=false 2023-02-21 09:57:27,123 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:27,126 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:27,126 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:27,126 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973447126"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973447126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973447126"}]},"ts":"1676973447126"} 2023-02-21 09:57:27,126 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:27,126 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447126"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973447126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973447126"}]},"ts":"1676973447126"} 2023-02-21 09:57:27,126 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447126"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973447126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973447126"}]},"ts":"1676973447126"} 2023-02-21 09:57:27,126 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:27,126 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:27,126 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447126"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973447126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973447126"}]},"ts":"1676973447126"} 2023-02-21 09:57:27,127 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973447126"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973447126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973447126"}]},"ts":"1676973447126"} 2023-02-21 09:57:27,128 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=208, state=RUNNABLE; OpenRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:27,128 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=207, state=RUNNABLE; OpenRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:27,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=206, state=RUNNABLE; OpenRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:27,129 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:27,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=203, state=RUNNABLE; OpenRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:27,130 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:27,130 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447130"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973447130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973447130"}]},"ts":"1676973447130"} 2023-02-21 09:57:27,131 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=204, state=RUNNABLE; OpenRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:27,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-21 09:57:27,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81b4d0d7d383dbd3ed3b6da2e1fab12e, NAME => 'testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6472b4dfac0a398ae332dd5f9936afab, NAME => 'testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ca9f39730d0c524ed83cab2e18c641c8, NAME => 'testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,283 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,284 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,284 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,285 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf 2023-02-21 09:57:27,285 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf 2023-02-21 09:57:27,285 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf 2023-02-21 09:57:27,285 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf 2023-02-21 09:57:27,285 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6472b4dfac0a398ae332dd5f9936afab columnFamilyName cf 2023-02-21 09:57:27,285 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81b4d0d7d383dbd3ed3b6da2e1fab12e columnFamilyName cf 2023-02-21 09:57:27,286 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf 2023-02-21 09:57:27,286 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf 2023-02-21 09:57:27,286 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca9f39730d0c524ed83cab2e18c641c8 columnFamilyName cf 2023-02-21 09:57:27,286 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(310): Store=6472b4dfac0a398ae332dd5f9936afab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:27,286 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(310): Store=81b4d0d7d383dbd3ed3b6da2e1fab12e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:27,287 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(310): Store=ca9f39730d0c524ed83cab2e18c641c8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:27,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:27,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:27,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:27,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:27,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:27,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:27,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6472b4dfac0a398ae332dd5f9936afab; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69430783, jitterRate=0.034599289298057556}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:27,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:27,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ca9f39730d0c524ed83cab2e18c641c8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72774605, jitterRate=0.08442611992359161}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:27,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 81b4d0d7d383dbd3ed3b6da2e1fab12e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71193168, jitterRate=0.06086087226867676}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:27,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:27,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:27,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e., pid=209, masterSystemTime=1676973447279 2023-02-21 09:57:27,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8., pid=211, masterSystemTime=1676973447280 2023-02-21 09:57:27,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab., pid=210, masterSystemTime=1676973447280 2023-02-21 09:57:27,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:27,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:27,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:27,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bdca281384ff4be099bfeae0c42d2c6, NAME => 'testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:27,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,296 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:27,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:27,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,296 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447295"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973447295"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973447295"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973447295"}]},"ts":"1676973447295"} 2023-02-21 09:57:27,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:27,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:27,296 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:27,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:27,296 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447296"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973447296"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973447296"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973447296"}]},"ts":"1676973447296"} 2023-02-21 09:57:27,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14df7364e6a8c8dc1107428e33ce230a, NAME => 'testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:27,297 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:27,297 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b51ce6c2c565e8d6b692d66c994428bb, NAME => 'testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,297 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:27,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,297 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:27,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,298 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973447297"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973447297"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973447297"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973447297"}]},"ts":"1676973447297"} 2023-02-21 09:57:27,298 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,299 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,300 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf 2023-02-21 09:57:27,300 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf 2023-02-21 09:57:27,300 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b51ce6c2c565e8d6b692d66c994428bb columnFamilyName cf 2023-02-21 09:57:27,300 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=207 2023-02-21 09:57:27,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=206 2023-02-21 09:57:27,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=207, state=SUCCESS; OpenRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584 in 169 msec 2023-02-21 09:57:27,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=206, state=SUCCESS; OpenRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614 in 169 msec 2023-02-21 09:57:27,301 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf 2023-02-21 09:57:27,301 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf 2023-02-21 09:57:27,301 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14df7364e6a8c8dc1107428e33ce230a columnFamilyName cf 2023-02-21 09:57:27,301 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(310): Store=b51ce6c2c565e8d6b692d66c994428bb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:27,302 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN in 330 msec 2023-02-21 09:57:27,302 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN in 331 msec 2023-02-21 09:57:27,302 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(310): Store=14df7364e6a8c8dc1107428e33ce230a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:27,302 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf 2023-02-21 09:57:27,302 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf 2023-02-21 09:57:27,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,303 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bdca281384ff4be099bfeae0c42d2c6 columnFamilyName cf 2023-02-21 09:57:27,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,303 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(310): Store=9bdca281384ff4be099bfeae0c42d2c6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:27,303 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=208 2023-02-21 09:57:27,304 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=208, state=SUCCESS; OpenRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539 in 175 msec 2023-02-21 09:57:27,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN in 334 msec 2023-02-21 09:57:27,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:27,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:27,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:27,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:27,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b51ce6c2c565e8d6b692d66c994428bb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60506871, jitterRate=-0.09837736189365387}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:27,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:27,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:27,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb., pid=213, masterSystemTime=1676973447279 2023-02-21 09:57:27,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14df7364e6a8c8dc1107428e33ce230a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67292290, jitterRate=0.0027332603931427}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:27,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:27,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a., pid=214, masterSystemTime=1676973447280 2023-02-21 09:57:27,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-21 09:57:27,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:27,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:27,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9bdca281384ff4be099bfeae0c42d2c6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73453622, jitterRate=0.09454426169395447}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:27,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:27,310 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:27,311 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973447310"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973447310"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973447310"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973447310"}]},"ts":"1676973447310"} 2023-02-21 09:57:27,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6., pid=212, masterSystemTime=1676973447280 2023-02-21 09:57:27,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:27,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:27,311 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:27,311 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447311"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973447311"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973447311"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973447311"}]},"ts":"1676973447311"} 2023-02-21 09:57:27,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:27,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:27,312 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:27,312 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973447312"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973447312"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973447312"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973447312"}]},"ts":"1676973447312"} 2023-02-21 09:57:27,314 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=203 2023-02-21 09:57:27,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=204 2023-02-21 09:57:27,314 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=203, state=SUCCESS; OpenRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539 in 182 msec 2023-02-21 09:57:27,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=204, state=SUCCESS; OpenRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614 in 181 msec 2023-02-21 09:57:27,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN in 344 msec 2023-02-21 09:57:27,315 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN in 344 msec 2023-02-21 09:57:27,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-21 09:57:27,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584 in 184 msec 2023-02-21 09:57:27,316 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=205, resume processing ppid=202 2023-02-21 09:57:27,316 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN in 345 msec 2023-02-21 09:57:27,317 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-21 09:57:27,317 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973447317"}]},"ts":"1676973447317"} 2023-02-21 09:57:27,318 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=ENABLED in hbase:meta 2023-02-21 09:57:27,329 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 execute state=CREATE_TABLE_POST_OPERATION 2023-02-21 09:57:27,330 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676973446507 in 820 msec 2023-02-21 09:57:28,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-21 09:57:28,015 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 202 completed 2023-02-21 09:57:28,017 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,017 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,020 INFO [Listener at localhost/40243] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-21 09:57:28,033 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,034 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-21 09:57:28,038 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973448037"}]},"ts":"1676973448037"} 2023-02-21 09:57:28,039 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=DISABLING in hbase:meta 2023-02-21 09:57:28,040 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676973446507 to state=DISABLING 2023-02-21 09:57:28,041 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN}] 2023-02-21 09:57:28,042 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN 2023-02-21 09:57:28,043 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN 2023-02-21 09:57:28,043 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN 2023-02-21 09:57:28,043 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN 2023-02-21 09:57:28,043 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN 2023-02-21 09:57:28,043 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,043 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,043 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,043 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,043 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,044 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448043"}]},"ts":"1676973448043"} 2023-02-21 09:57:28,044 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448043"}]},"ts":"1676973448043"} 2023-02-21 09:57:28,044 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448043"}]},"ts":"1676973448043"} 2023-02-21 09:57:28,044 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448043"}]},"ts":"1676973448043"} 2023-02-21 09:57:28,044 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448043"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448043"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448043"}]},"ts":"1676973448043"} 2023-02-21 09:57:28,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=220, state=RUNNABLE; CloseRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:28,045 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=219, state=RUNNABLE; CloseRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:28,046 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=217, state=RUNNABLE; CloseRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:28,047 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=218, state=RUNNABLE; CloseRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:28,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=221, state=RUNNABLE; CloseRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:28,047 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN 2023-02-21 09:57:28,048 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,048 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448048"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448048"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448048"}]},"ts":"1676973448048"} 2023-02-21 09:57:28,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:28,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9bdca281384ff4be099bfeae0c42d2c6, disabling compactions & flushes 2023-02-21 09:57:28,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. after waiting 0 ms 2023-02-21 09:57:28,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ca9f39730d0c524ed83cab2e18c641c8, disabling compactions & flushes 2023-02-21 09:57:28,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. after waiting 0 ms 2023-02-21 09:57:28,198 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b51ce6c2c565e8d6b692d66c994428bb, disabling compactions & flushes 2023-02-21 09:57:28,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. after waiting 0 ms 2023-02-21 09:57:28,199 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:28,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:28,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:28,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:28,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:28,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:28,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6472b4dfac0a398ae332dd5f9936afab, disabling compactions & flushes 2023-02-21 09:57:28,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. after waiting 0 ms 2023-02-21 09:57:28,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,203 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=CLOSED 2023-02-21 09:57:28,203 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448203"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973448203"}]},"ts":"1676973448203"} 2023-02-21 09:57:28,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14df7364e6a8c8dc1107428e33ce230a, disabling compactions & flushes 2023-02-21 09:57:28,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. after waiting 0 ms 2023-02-21 09:57:28,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,205 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=CLOSED 2023-02-21 09:57:28,205 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448205"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973448205"}]},"ts":"1676973448205"} 2023-02-21 09:57:28,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 81b4d0d7d383dbd3ed3b6da2e1fab12e, disabling compactions & flushes 2023-02-21 09:57:28,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. after waiting 0 ms 2023-02-21 09:57:28,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,206 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=CLOSED 2023-02-21 09:57:28,206 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448206"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973448206"}]},"ts":"1676973448206"} 2023-02-21 09:57:28,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:28,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:28,207 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=218 2023-02-21 09:57:28,207 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=218, state=SUCCESS; CloseRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584 in 158 msec 2023-02-21 09:57:28,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:28,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=219 2023-02-21 09:57:28,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=219, state=SUCCESS; CloseRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614 in 161 msec 2023-02-21 09:57:28,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-21 09:57:28,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:28,209 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN in 166 msec 2023-02-21 09:57:28,209 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=CLOSED 2023-02-21 09:57:28,209 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448209"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973448209"}]},"ts":"1676973448209"} 2023-02-21 09:57:28,209 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-21 09:57:28,209 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539 in 158 msec 2023-02-21 09:57:28,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:28,210 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN in 168 msec 2023-02-21 09:57:28,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN in 168 msec 2023-02-21 09:57:28,211 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=CLOSED 2023-02-21 09:57:28,211 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973448211"}]},"ts":"1676973448211"} 2023-02-21 09:57:28,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,212 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=CLOSED 2023-02-21 09:57:28,212 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448212"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973448212"}]},"ts":"1676973448212"} 2023-02-21 09:57:28,212 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=220 2023-02-21 09:57:28,212 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=220, state=SUCCESS; CloseRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584 in 166 msec 2023-02-21 09:57:28,214 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN in 172 msec 2023-02-21 09:57:28,214 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=217 2023-02-21 09:57:28,214 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=217, state=SUCCESS; CloseRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614 in 166 msec 2023-02-21 09:57:28,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=221 2023-02-21 09:57:28,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=221, state=SUCCESS; CloseRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539 in 166 msec 2023-02-21 09:57:28,215 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN in 173 msec 2023-02-21 09:57:28,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=221, resume processing ppid=215 2023-02-21 09:57:28,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN in 174 msec 2023-02-21 09:57:28,217 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973448217"}]},"ts":"1676973448217"} 2023-02-21 09:57:28,218 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=DISABLED in hbase:meta 2023-02-21 09:57:28,220 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676973446507 to state=DISABLED 2023-02-21 09:57:28,221 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676973446507 in 186 msec 2023-02-21 09:57:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-21 09:57:28,289 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 215 completed 2023-02-21 09:57:28,289 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } 2023-02-21 09:57:28,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973448289 (current time:1676973448289). 2023-02-21 09:57:28,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:28,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x7a1a82f3 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1116020d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:28,298 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41064, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:28,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a1a82f3 to 127.0.0.1:58856 2023-02-21 09:57:28,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:28,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:28,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } 2023-02-21 09:57:28,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE 2023-02-21 09:57:28,302 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE 2023-02-21 09:57:28,303 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE; timeout=600000, timestamp=1676974048303 2023-02-21 09:57:28,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } 2023-02-21 09:57:28,303 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676973446507 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,303 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:28,303 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:28,312 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=DISABLED ttl=0 } 2023-02-21 09:57:28,317 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,317 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,317 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,318 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,324 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:28,325 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,325 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,325 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,326 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,326 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,326 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,327 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-21 09:57:28,379 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:28,379 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676973446507 2023-02-21 09:57:28,380 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973446507 2023-02-21 09:57:28,395 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973446507 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/emptySnaptb-1676973446507 2023-02-21 09:57:28,412 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676973446507 of table testCorruptedSnapshot-1676973446507 completed 2023-02-21 09:57:28,412 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973446507 2023-02-21 09:57:28,412 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/emptySnaptb-1676973446507 2023-02-21 09:57:28,412 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676973446507 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676973446507 at 1676973448303Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=DISABLED ttl=0 } at 1676973448312 (+9 ms)Consolidate snapshot: emptySnaptb-1676973446507 at 1676973448379 (+67 ms)Loading Region manifests for emptySnaptb-1676973446507 at 1676973448380 (+1 ms)Writing data manifest for emptySnaptb-1676973446507 at 1676973448383 (+3 ms)Verifying snapshot: emptySnaptb-1676973446507 at 1676973448391 (+8 ms)Snapshot emptySnaptb-1676973446507 of table testCorruptedSnapshot-1676973446507 completed at 1676973448412 (+21 ms) 2023-02-21 09:57:28,415 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE 2023-02-21 09:57:28,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE in 114 msec 2023-02-21 09:57:28,554 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:28,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } is done 2023-02-21 09:57:28,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:28,555 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,555 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-21 09:57:28,559 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:28,559 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973448559"}]},"ts":"1676973448559"} 2023-02-21 09:57:28,560 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=ENABLING in hbase:meta 2023-02-21 09:57:28,562 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:28,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN}] 2023-02-21 09:57:28,563 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN 2023-02-21 09:57:28,563 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN 2023-02-21 09:57:28,563 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN 2023-02-21 09:57:28,563 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN 2023-02-21 09:57:28,564 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,564 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,564 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:28,564 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:28,564 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,564 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,564 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN 2023-02-21 09:57:28,564 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:28,564 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:28,565 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,565 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN 2023-02-21 09:57:28,565 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:28,565 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,565 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:28,714 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:28,714 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,714 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,714 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448714"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448714"}]},"ts":"1676973448714"} 2023-02-21 09:57:28,714 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,714 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,715 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448714"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448714"}]},"ts":"1676973448714"} 2023-02-21 09:57:28,714 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,715 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448714"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448714"}]},"ts":"1676973448714"} 2023-02-21 09:57:28,715 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448714"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448714"}]},"ts":"1676973448714"} 2023-02-21 09:57:28,714 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448714"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448714"}]},"ts":"1676973448714"} 2023-02-21 09:57:28,716 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=235, state=RUNNABLE; OpenRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:28,716 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=234, state=RUNNABLE; OpenRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:28,717 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=230, state=RUNNABLE; OpenRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:28,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=232, state=RUNNABLE; OpenRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:28,718 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=233, state=RUNNABLE; OpenRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:28,719 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,719 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448719"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973448719"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973448719"}]},"ts":"1676973448719"} 2023-02-21 09:57:28,720 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=231, state=RUNNABLE; OpenRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-21 09:57:28,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81b4d0d7d383dbd3ed3b6da2e1fab12e, NAME => 'testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:28,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6472b4dfac0a398ae332dd5f9936afab, NAME => 'testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,872 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,872 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,872 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf 2023-02-21 09:57:28,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ca9f39730d0c524ed83cab2e18c641c8, NAME => 'testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:28,873 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf 2023-02-21 09:57:28,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:28,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,873 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81b4d0d7d383dbd3ed3b6da2e1fab12e columnFamilyName cf 2023-02-21 09:57:28,873 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf 2023-02-21 09:57:28,873 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf 2023-02-21 09:57:28,874 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6472b4dfac0a398ae332dd5f9936afab columnFamilyName cf 2023-02-21 09:57:28,874 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(310): Store=81b4d0d7d383dbd3ed3b6da2e1fab12e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:28,874 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,874 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(310): Store=6472b4dfac0a398ae332dd5f9936afab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:28,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,875 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf 2023-02-21 09:57:28,875 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf 2023-02-21 09:57:28,875 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca9f39730d0c524ed83cab2e18c641c8 columnFamilyName cf 2023-02-21 09:57:28,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,876 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(310): Store=ca9f39730d0c524ed83cab2e18c641c8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:28,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:28,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:28,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 81b4d0d7d383dbd3ed3b6da2e1fab12e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73192163, jitterRate=0.09064821898937225}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:28,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:28,879 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6472b4dfac0a398ae332dd5f9936afab; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62078649, jitterRate=-0.07495604455471039}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:28,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:28,879 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e., pid=236, masterSystemTime=1676973448867 2023-02-21 09:57:28,879 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab., pid=237, masterSystemTime=1676973448868 2023-02-21 09:57:28,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:28,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:28,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ca9f39730d0c524ed83cab2e18c641c8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69549896, jitterRate=0.03637421131134033}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b51ce6c2c565e8d6b692d66c994428bb, NAME => 'testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:28,881 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,881 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448881"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973448881"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973448881"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973448881"}]},"ts":"1676973448881"} 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:28,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bdca281384ff4be099bfeae0c42d2c6, NAME => 'testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:28,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8., pid=240, masterSystemTime=1676973448871 2023-02-21 09:57:28,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:28,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,882 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,882 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,882 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448882"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973448882"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973448882"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973448882"}]},"ts":"1676973448882"} 2023-02-21 09:57:28,883 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf 2023-02-21 09:57:28,883 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf 2023-02-21 09:57:28,883 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,883 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b51ce6c2c565e8d6b692d66c994428bb columnFamilyName cf 2023-02-21 09:57:28,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:28,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14df7364e6a8c8dc1107428e33ce230a, NAME => 'testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:28,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:28,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,884 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(310): Store=b51ce6c2c565e8d6b692d66c994428bb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:28,885 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,885 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448885"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973448885"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973448885"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973448885"}]},"ts":"1676973448885"} 2023-02-21 09:57:28,885 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf 2023-02-21 09:57:28,885 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf 2023-02-21 09:57:28,885 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,885 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bdca281384ff4be099bfeae0c42d2c6 columnFamilyName cf 2023-02-21 09:57:28,886 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=235 2023-02-21 09:57:28,886 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=235, state=SUCCESS; OpenRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539 in 167 msec 2023-02-21 09:57:28,886 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf 2023-02-21 09:57:28,886 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf 2023-02-21 09:57:28,886 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14df7364e6a8c8dc1107428e33ce230a columnFamilyName cf 2023-02-21 09:57:28,887 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=234 2023-02-21 09:57:28,887 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=234, state=SUCCESS; OpenRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584 in 168 msec 2023-02-21 09:57:28,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,886 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(310): Store=9bdca281384ff4be099bfeae0c42d2c6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:28,887 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(310): Store=14df7364e6a8c8dc1107428e33ce230a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:28,887 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN in 324 msec 2023-02-21 09:57:28,888 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN in 325 msec 2023-02-21 09:57:28,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,888 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=233 2023-02-21 09:57:28,888 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=233, state=SUCCESS; OpenRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614 in 168 msec 2023-02-21 09:57:28,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,889 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN in 326 msec 2023-02-21 09:57:28,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:28,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b51ce6c2c565e8d6b692d66c994428bb; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64479778, jitterRate=-0.03917643427848816}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:28,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:28,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:28,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:28,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb., pid=238, masterSystemTime=1676973448867 2023-02-21 09:57:28,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9bdca281384ff4be099bfeae0c42d2c6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65257525, jitterRate=-0.027587100863456726}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:28,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14df7364e6a8c8dc1107428e33ce230a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74853203, jitterRate=0.11539964377880096}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:28,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:28,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:28,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6., pid=239, masterSystemTime=1676973448868 2023-02-21 09:57:28,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a., pid=241, masterSystemTime=1676973448871 2023-02-21 09:57:28,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:28,894 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:28,894 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973448894"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973448894"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973448894"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973448894"}]},"ts":"1676973448894"} 2023-02-21 09:57:28,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:28,895 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:28,895 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448895"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973448895"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973448895"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973448895"}]},"ts":"1676973448895"} 2023-02-21 09:57:28,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:28,895 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:28,896 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973448895"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973448895"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973448895"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973448895"}]},"ts":"1676973448895"} 2023-02-21 09:57:28,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=230 2023-02-21 09:57:28,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=230, state=SUCCESS; OpenRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539 in 178 msec 2023-02-21 09:57:28,898 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=232 2023-02-21 09:57:28,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=231 2023-02-21 09:57:28,898 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=232, state=SUCCESS; OpenRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584 in 179 msec 2023-02-21 09:57:28,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=231, state=SUCCESS; OpenRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614 in 177 msec 2023-02-21 09:57:28,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN in 336 msec 2023-02-21 09:57:28,899 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN in 336 msec 2023-02-21 09:57:28,900 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=232, resume processing ppid=229 2023-02-21 09:57:28,900 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN in 336 msec 2023-02-21 09:57:28,900 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973448900"}]},"ts":"1676973448900"} 2023-02-21 09:57:28,901 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=ENABLED in hbase:meta 2023-02-21 09:57:28,903 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676973446507' was successfully enabled. 2023-02-21 09:57:28,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676973446507 in 349 msec 2023-02-21 09:57:29,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-21 09:57:29,310 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 229 completed 2023-02-21 09:57:29,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:29,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:29,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:29,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:29,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:29,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:29,322 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:29,323 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:29,323 INFO [Listener at localhost/40243] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-21 09:57:29,333 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:29,334 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:29,335 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:29,336 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:29,336 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:29,339 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676973446507 2023-02-21 09:57:29,339 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676973446507 2023-02-21 09:57:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-21 09:57:29,343 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973449343"}]},"ts":"1676973449343"} 2023-02-21 09:57:29,344 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=DISABLING in hbase:meta 2023-02-21 09:57:29,346 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676973446507 to state=DISABLING 2023-02-21 09:57:29,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN}] 2023-02-21 09:57:29,347 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN 2023-02-21 09:57:29,347 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN 2023-02-21 09:57:29,347 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN 2023-02-21 09:57:29,347 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN 2023-02-21 09:57:29,348 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN 2023-02-21 09:57:29,348 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:29,348 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973449348"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973449348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973449348"}]},"ts":"1676973449348"} 2023-02-21 09:57:29,348 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:29,348 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:29,348 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449348"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973449348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973449348"}]},"ts":"1676973449348"} 2023-02-21 09:57:29,348 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449348"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973449348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973449348"}]},"ts":"1676973449348"} 2023-02-21 09:57:29,348 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:29,348 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:29,349 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449348"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973449348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973449348"}]},"ts":"1676973449348"} 2023-02-21 09:57:29,349 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449348"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973449348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973449348"}]},"ts":"1676973449348"} 2023-02-21 09:57:29,349 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=248, state=RUNNABLE; CloseRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:29,350 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=245, state=RUNNABLE; CloseRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:29,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=246, state=RUNNABLE; CloseRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:29,351 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=247, state=RUNNABLE; CloseRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:29,352 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN 2023-02-21 09:57:29,352 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=244, state=RUNNABLE; CloseRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:29,353 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:29,353 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973449353"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973449353"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973449353"}]},"ts":"1676973449353"} 2023-02-21 09:57:29,354 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:29,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:29,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:29,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 81b4d0d7d383dbd3ed3b6da2e1fab12e, disabling compactions & flushes 2023-02-21 09:57:29,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6472b4dfac0a398ae332dd5f9936afab, disabling compactions & flushes 2023-02-21 09:57:29,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:29,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:29,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:29,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:29,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. after waiting 0 ms 2023-02-21 09:57:29,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:29,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. after waiting 0 ms 2023-02-21 09:57:29,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:29,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6472b4dfac0a398ae332dd5f9936afab 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-21 09:57:29,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 81b4d0d7d383dbd3ed3b6da2e1fab12e 1/1 column families, dataSize=23.49 KB heapSize=50.73 KB 2023-02-21 09:57:29,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:29,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ca9f39730d0c524ed83cab2e18c641c8, disabling compactions & flushes 2023-02-21 09:57:29,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:29,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:29,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. after waiting 0 ms 2023-02-21 09:57:29,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:29,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ca9f39730d0c524ed83cab2e18c641c8 1/1 column families, dataSize=1.44 KB heapSize=3.34 KB 2023-02-21 09:57:29,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023022143c4a23bba144669a4959b54289d98ea to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c2023022143c4a23bba144669a4959b54289d98ea 2023-02-21 09:57:29,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230221fb90d0632ed14a6cac03f10e01135c01 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221fb90d0632ed14a6cac03f10e01135c01 2023-02-21 09:57:29,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/.tmp/cf/d06c68a1261349fdaea54214d1f6adfa 2023-02-21 09:57:29,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/.tmp/cf/8b29a5f1653247d09b1f72bfd86ddf1e 2023-02-21 09:57:29,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/.tmp/cf/d06c68a1261349fdaea54214d1f6adfa as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa 2023-02-21 09:57:29,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/.tmp/cf/8b29a5f1653247d09b1f72bfd86ddf1e as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e 2023-02-21 09:57:29,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa, entries=31, sequenceid=8, filesize=10.2 K 2023-02-21 09:57:29,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 6472b4dfac0a398ae332dd5f9936afab in 39ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:29,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676973446507' 2023-02-21 09:57:29,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e, entries=22, sequenceid=8, filesize=8.7 K 2023-02-21 09:57:29,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.44 KB/1472, heapSize ~3.33 KB/3408, currentSize=0 B/0 for ca9f39730d0c524ed83cab2e18c641c8 in 40ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:29,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:29,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:29,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:29,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:29,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:29,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:29,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9bdca281384ff4be099bfeae0c42d2c6, disabling compactions & flushes 2023-02-21 09:57:29,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:29,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:29,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:29,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. after waiting 0 ms 2023-02-21 09:57:29,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:29,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:29,548 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=CLOSED 2023-02-21 09:57:29,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9bdca281384ff4be099bfeae0c42d2c6 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-21 09:57:29,548 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449548"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973449548"}]},"ts":"1676973449548"} 2023-02-21 09:57:29,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:29,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:29,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14df7364e6a8c8dc1107428e33ce230a, disabling compactions & flushes 2023-02-21 09:57:29,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:29,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:29,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. after waiting 0 ms 2023-02-21 09:57:29,550 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=CLOSED 2023-02-21 09:57:29,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:29,550 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973449550"}]},"ts":"1676973449550"} 2023-02-21 09:57:29,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 14df7364e6a8c8dc1107428e33ce230a 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-21 09:57:29,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=247 2023-02-21 09:57:29,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=247, state=SUCCESS; CloseRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584 in 199 msec 2023-02-21 09:57:29,553 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN in 206 msec 2023-02-21 09:57:29,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=246 2023-02-21 09:57:29,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=246, state=SUCCESS; CloseRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614 in 202 msec 2023-02-21 09:57:29,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN in 208 msec 2023-02-21 09:57:29,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230221df3103c8480844a18c55fe1d917da6f3 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221df3103c8480844a18c55fe1d917da6f3 2023-02-21 09:57:29,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230221d1b95928585f4161bd7f1160338f01eb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b20230221d1b95928585f4161bd7f1160338f01eb 2023-02-21 09:57:29,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/.tmp/cf/1e36358cbb014a39be0ea2aee7b17873 2023-02-21 09:57:29,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/.tmp/cf/1351f6c79fc84963b20638e4e7fe0618 2023-02-21 09:57:29,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/.tmp/cf/1e36358cbb014a39be0ea2aee7b17873 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873 2023-02-21 09:57:29,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/.tmp/cf/1351f6c79fc84963b20638e4e7fe0618 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618 2023-02-21 09:57:29,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873, entries=31, sequenceid=8, filesize=10.2 K 2023-02-21 09:57:29,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 9bdca281384ff4be099bfeae0c42d2c6 in 37ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:29,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618, entries=26, sequenceid=8, filesize=9.4 K 2023-02-21 09:57:29,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 14df7364e6a8c8dc1107428e33ce230a in 36ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:29,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:29,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:29,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:29,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:29,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:29,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:29,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:29,591 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=CLOSED 2023-02-21 09:57:29,591 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449591"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973449591"}]},"ts":"1676973449591"} 2023-02-21 09:57:29,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:29,592 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=CLOSED 2023-02-21 09:57:29,592 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973449592"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973449592"}]},"ts":"1676973449592"} 2023-02-21 09:57:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-21 09:57:29,594 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=244 2023-02-21 09:57:29,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=244, state=SUCCESS; CloseRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614 in 241 msec 2023-02-21 09:57:29,595 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=245 2023-02-21 09:57:29,595 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=245, state=SUCCESS; CloseRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584 in 243 msec 2023-02-21 09:57:29,596 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN in 248 msec 2023-02-21 09:57:29,596 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN in 249 msec 2023-02-21 09:57:29,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230221afee0ccb4b734defb1ba85583706faec to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d520230221afee0ccb4b734defb1ba85583706faec 2023-02-21 09:57:29,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/.tmp/cf/f6a6cebecebc42b7920526e24f376d47 2023-02-21 09:57:29,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/.tmp/cf/f6a6cebecebc42b7920526e24f376d47 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47 2023-02-21 09:57:29,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47, entries=359, sequenceid=8, filesize=66.7 K 2023-02-21 09:57:29,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.49 KB/24049, heapSize ~50.72 KB/51936, currentSize=0 B/0 for 81b4d0d7d383dbd3ed3b6da2e1fab12e in 441ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:29,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:29,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:29,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:29,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:29,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:29,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b51ce6c2c565e8d6b692d66c994428bb, disabling compactions & flushes 2023-02-21 09:57:29,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:29,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:29,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. after waiting 0 ms 2023-02-21 09:57:29,947 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=CLOSED 2023-02-21 09:57:29,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:29,947 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973449947"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973449947"}]},"ts":"1676973449947"} 2023-02-21 09:57:29,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b51ce6c2c565e8d6b692d66c994428bb 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-21 09:57:29,952 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=248 2023-02-21 09:57:29,952 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=248, state=SUCCESS; CloseRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539 in 602 msec 2023-02-21 09:57:29,953 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN in 606 msec 2023-02-21 09:57:29,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302211803b10023214dca9b99ba16f2b656cb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e202302211803b10023214dca9b99ba16f2b656cb 2023-02-21 09:57:29,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/.tmp/cf/eb37b03d9d2c471e9586d329a55957b9 2023-02-21 09:57:29,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/.tmp/cf/eb37b03d9d2c471e9586d329a55957b9 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9 2023-02-21 09:57:29,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9, entries=31, sequenceid=8, filesize=10.2 K 2023-02-21 09:57:29,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for b51ce6c2c565e8d6b692d66c994428bb in 32ms, sequenceid=8, compaction requested=false 2023-02-21 09:57:29,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-21 09:57:29,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:29,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:29,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:29,983 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=CLOSED 2023-02-21 09:57:29,983 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973449983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973449983"}]},"ts":"1676973449983"} 2023-02-21 09:57:29,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-21 09:57:29,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539 in 630 msec 2023-02-21 09:57:29,986 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=243, resume processing ppid=242 2023-02-21 09:57:29,986 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN in 639 msec 2023-02-21 09:57:29,987 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973449987"}]},"ts":"1676973449987"} 2023-02-21 09:57:29,988 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=DISABLED in hbase:meta 2023-02-21 09:57:29,990 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676973446507 to state=DISABLED 2023-02-21 09:57:29,991 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676973446507 in 650 msec 2023-02-21 09:57:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-21 09:57:30,095 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 242 completed 2023-02-21 09:57:30,096 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } 2023-02-21 09:57:30,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676973450096 (current time:1676973450096). 2023-02-21 09:57:30,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-21 09:57:30,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(139): Connect 0x2e163970 to 127.0.0.1:58856 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-21 09:57:30,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@353b5b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-21 09:57:30,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-21 09:57:30,105 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41068, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-21 09:57:30,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e163970 to 127.0.0.1:58856 2023-02-21 09:57:30,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:30,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-21 09:57:30,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } 2023-02-21 09:57:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE 2023-02-21 09:57:30,109 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE 2023-02-21 09:57:30,110 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE; timeout=600000, timestamp=1676974050110 2023-02-21 09:57:30,110 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676973446507 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } 2023-02-21 09:57:30,110 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-21 09:57:30,110 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-21 09:57:30,125 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=DISABLED ttl=0 } 2023-02-21 09:57:30,126 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,126 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,126 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,126 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-21 09:57:30,127 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-21 09:57:30,128 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa] hfiles 2023-02-21 09:57:30,128 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618] hfiles 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9] hfiles 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47] hfiles 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873] hfiles 2023-02-21 09:57:30,129 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873 2023-02-21 09:57:30,130 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e] hfiles 2023-02-21 09:57:30,130 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c2023022143c4a23bba144669a4959b54289d98ea, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b20230221d1b95928585f4161bd7f1160338f01eb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221df3103c8480844a18c55fe1d917da6f3, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e202302211803b10023214dca9b99ba16f2b656cb, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d520230221afee0ccb4b734defb1ba85583706faec, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221fb90d0632ed14a6cac03f10e01135c01] hfiles 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c2023022143c4a23bba144669a4959b54289d98ea 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b20230221d1b95928585f4161bd7f1160338f01eb 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221df3103c8480844a18c55fe1d917da6f3 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e202302211803b10023214dca9b99ba16f2b656cb 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d520230221afee0ccb4b734defb1ba85583706faec 2023-02-21 09:57:30,134 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221fb90d0632ed14a6cac03f10e01135c01 2023-02-21 09:57:30,152 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=DISABLED ttl=0 } as finished. 2023-02-21 09:57:30,152 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676973446507 2023-02-21 09:57:30,153 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973446507 2023-02-21 09:57:30,171 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973446507 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/snaptb0-1676973446507 2023-02-21 09:57:30,186 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676973446507 of table testCorruptedSnapshot-1676973446507 completed 2023-02-21 09:57:30,186 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973446507 2023-02-21 09:57:30,187 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.hbase-snapshot/.tmp/snaptb0-1676973446507 2023-02-21 09:57:30,187 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676973446507 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676973446507 at 1676973450110Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=DISABLED ttl=0 } at 1676973450125 (+15 ms)Consolidate snapshot: snaptb0-1676973446507 at 1676973450152 (+27 ms)Loading Region manifests for snaptb0-1676973446507 at 1676973450153 (+1 ms)Writing data manifest for snaptb0-1676973446507 at 1676973450156 (+3 ms)Verifying snapshot: snaptb0-1676973446507 at 1676973450164 (+8 ms)Snapshot snaptb0-1676973446507 of table testCorruptedSnapshot-1676973446507 completed at 1676973450186 (+22 ms) 2023-02-21 09:57:30,188 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE 2023-02-21 09:57:30,189 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676973446507, type=EXCLUSIVE in 80 msec 2023-02-21 09:57:30,360 DEBUG [Listener at localhost/40243] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-21 09:57:30,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 } is done 2023-02-21 09:57:30,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676973446507 table=testCorruptedSnapshot-1676973446507 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-21 09:57:30,362 INFO [Listener at localhost/40243] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676973446507 2023-02-21 09:57:30,362 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676973446507 2023-02-21 09:57:30,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:30,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-21 09:57:30,365 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:30,366 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973450366"}]},"ts":"1676973450366"} 2023-02-21 09:57:30,367 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=ENABLING in hbase:meta 2023-02-21 09:57:30,369 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-21 09:57:30,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN}] 2023-02-21 09:57:30,370 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN 2023-02-21 09:57:30,370 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN 2023-02-21 09:57:30,370 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN 2023-02-21 09:57:30,371 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN 2023-02-21 09:57:30,371 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:30,371 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN 2023-02-21 09:57:30,371 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:30,371 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:30,371 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:30,371 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:30,371 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:30,371 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41011,1676973419614; forceNewPlan=false, retain=true 2023-02-21 09:57:30,371 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44835,1676973419584; forceNewPlan=false, retain=true 2023-02-21 09:57:30,372 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:30,372 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN 2023-02-21 09:57:30,372 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:30,373 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:30,373 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46077,1676973419539; forceNewPlan=false, retain=true 2023-02-21 09:57:30,521 INFO [jenkins-hbase4:41687] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-21 09:57:30,521 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:30,521 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:30,521 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:30,522 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450521"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973450521"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973450521"}]},"ts":"1676973450521"} 2023-02-21 09:57:30,522 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973450521"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973450521"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973450521"}]},"ts":"1676973450521"} 2023-02-21 09:57:30,521 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:30,521 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:30,522 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973450521"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973450521"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973450521"}]},"ts":"1676973450521"} 2023-02-21 09:57:30,522 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450521"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973450521"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973450521"}]},"ts":"1676973450521"} 2023-02-21 09:57:30,522 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450521"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973450521"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973450521"}]},"ts":"1676973450521"} 2023-02-21 09:57:30,523 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=260, state=RUNNABLE; OpenRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:30,524 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=257, state=RUNNABLE; OpenRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:30,524 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=262, state=RUNNABLE; OpenRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:30,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=261, state=RUNNABLE; OpenRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:30,526 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=259, state=RUNNABLE; OpenRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:30,527 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:30,527 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450527"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973450527"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973450527"}]},"ts":"1676973450527"} 2023-02-21 09:57:30,528 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=258, state=RUNNABLE; OpenRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:30,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-21 09:57:30,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:30,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ca9f39730d0c524ed83cab2e18c641c8, NAME => 'testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.', STARTKEY => '3', ENDKEY => '4'} 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b51ce6c2c565e8d6b692d66c994428bb, NAME => 'testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.', STARTKEY => '', ENDKEY => '1'} 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:30,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6472b4dfac0a398ae332dd5f9936afab, NAME => 'testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.', STARTKEY => '4', ENDKEY => '5'} 2023-02-21 09:57:30,679 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,679 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:30,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,680 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf 2023-02-21 09:57:30,680 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf 2023-02-21 09:57:30,680 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf 2023-02-21 09:57:30,680 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf 2023-02-21 09:57:30,681 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,681 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca9f39730d0c524ed83cab2e18c641c8 columnFamilyName cf 2023-02-21 09:57:30,681 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b51ce6c2c565e8d6b692d66c994428bb columnFamilyName cf 2023-02-21 09:57:30,681 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf 2023-02-21 09:57:30,681 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf 2023-02-21 09:57:30,681 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6472b4dfac0a398ae332dd5f9936afab columnFamilyName cf 2023-02-21 09:57:30,684 DEBUG [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e 2023-02-21 09:57:30,684 DEBUG [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9 2023-02-21 09:57:30,685 INFO [StoreOpener-ca9f39730d0c524ed83cab2e18c641c8-1] regionserver.HStore(310): Store=ca9f39730d0c524ed83cab2e18c641c8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:30,685 INFO [StoreOpener-b51ce6c2c565e8d6b692d66c994428bb-1] regionserver.HStore(310): Store=b51ce6c2c565e8d6b692d66c994428bb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:30,685 DEBUG [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa 2023-02-21 09:57:30,685 INFO [StoreOpener-6472b4dfac0a398ae332dd5f9936afab-1] regionserver.HStore(310): Store=6472b4dfac0a398ae332dd5f9936afab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:30,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:30,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:30,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:30,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b51ce6c2c565e8d6b692d66c994428bb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74527854, jitterRate=0.11055156588554382}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:30,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:30,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ca9f39730d0c524ed83cab2e18c641c8; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64747117, jitterRate=-0.03519277274608612}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:30,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:30,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb., pid=264, masterSystemTime=1676973450675 2023-02-21 09:57:30,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6472b4dfac0a398ae332dd5f9936afab; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62459986, jitterRate=-0.0692736804485321}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:30,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:30,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8., pid=263, masterSystemTime=1676973450675 2023-02-21 09:57:30,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab., pid=266, masterSystemTime=1676973450677 2023-02-21 09:57:30,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:30,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:30,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:30,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81b4d0d7d383dbd3ed3b6da2e1fab12e, NAME => 'testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.', STARTKEY => '5', ENDKEY => ''} 2023-02-21 09:57:30,691 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:30,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,692 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973450691"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973450691"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973450691"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973450691"}]},"ts":"1676973450691"} 2023-02-21 09:57:30,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:30,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:30,692 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 14df7364e6a8c8dc1107428e33ce230a, NAME => 'testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.', STARTKEY => '1', ENDKEY => '2'} 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:30,692 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450692"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973450692"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973450692"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973450692"}]},"ts":"1676973450692"} 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:30,693 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:30,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,693 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450693"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973450693"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973450693"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973450693"}]},"ts":"1676973450693"} 2023-02-21 09:57:30,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:30,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bdca281384ff4be099bfeae0c42d2c6, NAME => 'testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.', STARTKEY => '2', ENDKEY => '3'} 2023-02-21 09:57:30,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676973446507 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-21 09:57:30,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,694 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=257 2023-02-21 09:57:30,694 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=257, state=SUCCESS; OpenRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539 in 169 msec 2023-02-21 09:57:30,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, ASSIGN in 325 msec 2023-02-21 09:57:30,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=260 2023-02-21 09:57:30,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=260, state=SUCCESS; OpenRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614 in 171 msec 2023-02-21 09:57:30,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=261 2023-02-21 09:57:30,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=261, state=SUCCESS; OpenRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584 in 169 msec 2023-02-21 09:57:30,696 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, ASSIGN in 326 msec 2023-02-21 09:57:30,697 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, ASSIGN in 326 msec 2023-02-21 09:57:30,698 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,698 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,699 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,699 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf 2023-02-21 09:57:30,699 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf 2023-02-21 09:57:30,699 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf 2023-02-21 09:57:30,700 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf 2023-02-21 09:57:30,700 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf 2023-02-21 09:57:30,700 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf 2023-02-21 09:57:30,700 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 14df7364e6a8c8dc1107428e33ce230a columnFamilyName cf 2023-02-21 09:57:30,700 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81b4d0d7d383dbd3ed3b6da2e1fab12e columnFamilyName cf 2023-02-21 09:57:30,700 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bdca281384ff4be099bfeae0c42d2c6 columnFamilyName cf 2023-02-21 09:57:30,705 DEBUG [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618 2023-02-21 09:57:30,705 DEBUG [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47 2023-02-21 09:57:30,705 INFO [StoreOpener-14df7364e6a8c8dc1107428e33ce230a-1] regionserver.HStore(310): Store=14df7364e6a8c8dc1107428e33ce230a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:30,705 INFO [StoreOpener-81b4d0d7d383dbd3ed3b6da2e1fab12e-1] regionserver.HStore(310): Store=81b4d0d7d383dbd3ed3b6da2e1fab12e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:30,706 DEBUG [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(539): loaded hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873 2023-02-21 09:57:30,706 INFO [StoreOpener-9bdca281384ff4be099bfeae0c42d2c6-1] regionserver.HStore(310): Store=9bdca281384ff4be099bfeae0c42d2c6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-21 09:57:30,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:30,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:30,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 14df7364e6a8c8dc1107428e33ce230a; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68778976, jitterRate=0.024886608123779297}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:30,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:30,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:30,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 81b4d0d7d383dbd3ed3b6da2e1fab12e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68761306, jitterRate=0.02462330460548401}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:30,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:30,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a., pid=268, masterSystemTime=1676973450675 2023-02-21 09:57:30,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9bdca281384ff4be099bfeae0c42d2c6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69318031, jitterRate=0.032919153571128845}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-21 09:57:30,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:30,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e., pid=265, masterSystemTime=1676973450675 2023-02-21 09:57:30,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6., pid=267, masterSystemTime=1676973450677 2023-02-21 09:57:30,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:30,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:30,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:30,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:30,716 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:30,716 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:30,716 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973450716"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973450716"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973450716"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973450716"}]},"ts":"1676973450716"} 2023-02-21 09:57:30,716 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450716"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973450716"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973450716"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973450716"}]},"ts":"1676973450716"} 2023-02-21 09:57:30,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:30,716 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:30,717 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:30,717 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973450717"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676973450717"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676973450717"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676973450717"}]},"ts":"1676973450717"} 2023-02-21 09:57:30,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=262 2023-02-21 09:57:30,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=259 2023-02-21 09:57:30,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=262, state=SUCCESS; OpenRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539 in 193 msec 2023-02-21 09:57:30,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=259, state=SUCCESS; OpenRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584 in 192 msec 2023-02-21 09:57:30,722 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=258 2023-02-21 09:57:30,722 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=258, state=SUCCESS; OpenRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614 in 191 msec 2023-02-21 09:57:30,722 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, ASSIGN in 351 msec 2023-02-21 09:57:30,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, ASSIGN in 351 msec 2023-02-21 09:57:30,723 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=258, resume processing ppid=256 2023-02-21 09:57:30,723 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, ASSIGN in 352 msec 2023-02-21 09:57:30,724 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973450724"}]},"ts":"1676973450724"} 2023-02-21 09:57:30,725 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=ENABLED in hbase:meta 2023-02-21 09:57:30,727 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676973446507' was successfully enabled. 2023-02-21 09:57:30,728 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676973446507 in 365 msec 2023-02-21 09:57:31,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-21 09:57:31,117 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 256 completed 2023-02-21 09:57:31,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:31,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:31,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41011] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:31,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44835] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:31,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:31,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46077] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-21 09:57:31,129 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676973446507 2023-02-21 09:57:31,130 DEBUG [Listener at localhost/40243] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:31,130 INFO [Listener at localhost/40243] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-21 09:57:31,226 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-21 09:57:31,287 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-21 09:57:31,357 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-21 09:57:31,408 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-21 09:57:31,464 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4672389c to 127.0.0.1:58856 2023-02-21 09:57:31,464 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:31,503 DEBUG [Listener at localhost/40243] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-21 09:57:32,316 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 1351f6c79fc84963b20638e4e7fe0618 in the real (hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618) or archive (hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618) directory for the primary table. 2023-02-21 09:57:32,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 1351f6c79fc84963b20638e4e7fe0618 in the real (hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618) or archive (hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2506) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2502) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-21 09:57:32,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] ipc.CallRunner(144): callId: 360 service: MasterService methodName: RestoreSnapshot size: 116 connection: 172.31.14.131:43920 deadline: 1676973512312, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 1351f6c79fc84963b20638e4e7fe0618 in the real (hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618) or archive (hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/1351f6c79fc84963b20638e4e7fe0618) directory for the primary table. 2023-02-21 09:57:32,321 INFO [Listener at localhost/40243] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676973446507 2023-02-21 09:57:32,321 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676973446507 2023-02-21 09:57:32,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:32,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-21 09:57:32,324 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973452324"}]},"ts":"1676973452324"} 2023-02-21 09:57:32,325 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=DISABLING in hbase:meta 2023-02-21 09:57:32,327 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676973446507 to state=DISABLING 2023-02-21 09:57:32,328 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN}] 2023-02-21 09:57:32,328 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN 2023-02-21 09:57:32,328 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN 2023-02-21 09:57:32,328 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN 2023-02-21 09:57:32,328 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN 2023-02-21 09:57:32,329 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN 2023-02-21 09:57:32,329 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:32,329 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:32,329 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452329"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973452329"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973452329"}]},"ts":"1676973452329"} 2023-02-21 09:57:32,329 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452329"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973452329"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973452329"}]},"ts":"1676973452329"} 2023-02-21 09:57:32,329 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:32,329 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452329"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973452329"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973452329"}]},"ts":"1676973452329"} 2023-02-21 09:57:32,330 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:32,330 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:32,330 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452330"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973452330"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973452330"}]},"ts":"1676973452330"} 2023-02-21 09:57:32,330 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973452330"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973452330"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973452330"}]},"ts":"1676973452330"} 2023-02-21 09:57:32,330 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=275, state=RUNNABLE; CloseRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:32,331 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=273, state=RUNNABLE; CloseRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584}] 2023-02-21 09:57:32,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=272, state=RUNNABLE; CloseRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:32,332 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=274, state=RUNNABLE; CloseRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614}] 2023-02-21 09:57:32,332 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=276, state=RUNNABLE; CloseRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:32,333 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN 2023-02-21 09:57:32,333 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:32,334 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973452333"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676973452333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676973452333"}]},"ts":"1676973452333"} 2023-02-21 09:57:32,334 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539}] 2023-02-21 09:57:32,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:32,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6472b4dfac0a398ae332dd5f9936afab, disabling compactions & flushes 2023-02-21 09:57:32,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:32,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:32,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. after waiting 0 ms 2023-02-21 09:57:32,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:32,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6472b4dfac0a398ae332dd5f9936afab 1/1 column families, dataSize=2.68 KB heapSize=6.02 KB 2023-02-21 09:57:32,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:32,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ca9f39730d0c524ed83cab2e18c641c8, disabling compactions & flushes 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b51ce6c2c565e8d6b692d66c994428bb, disabling compactions & flushes 2023-02-21 09:57:32,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:32,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. after waiting 0 ms 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. after waiting 0 ms 2023-02-21 09:57:32,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:32,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ca9f39730d0c524ed83cab2e18c641c8 1/1 column families, dataSize=1.24 KB heapSize=2.92 KB 2023-02-21 09:57:32,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b51ce6c2c565e8d6b692d66c994428bb 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-21 09:57:32,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302213db7cd893bd6460d89d9928822989130 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c202302213db7cd893bd6460d89d9928822989130 2023-02-21 09:57:32,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230221e7829692f3e44783ad2e74eb248fcf41 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221e7829692f3e44783ad2e74eb248fcf41 2023-02-21 09:57:32,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023022116b5057424584fa4b1e93ea14181e870 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e2023022116b5057424584fa4b1e93ea14181e870 2023-02-21 09:57:32,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/.tmp/cf/a2072e3f8aad440c865a4c001b6b7eb1 2023-02-21 09:57:32,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/.tmp/cf/2985652bfabb423ab0d21d9ad5945cdd 2023-02-21 09:57:32,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/.tmp/cf/a2072e3f8aad440c865a4c001b6b7eb1 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/a2072e3f8aad440c865a4c001b6b7eb1 2023-02-21 09:57:32,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/.tmp/cf/2985652bfabb423ab0d21d9ad5945cdd as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/2985652bfabb423ab0d21d9ad5945cdd 2023-02-21 09:57:32,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/.tmp/cf/8890a75f6e2c4a88984820bc67655763 2023-02-21 09:57:32,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/a2072e3f8aad440c865a4c001b6b7eb1, entries=41, sequenceid=15, filesize=12.0 K 2023-02-21 09:57:32,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.68 KB/2745, heapSize ~6 KB/6144, currentSize=0 B/0 for 6472b4dfac0a398ae332dd5f9936afab in 35ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:32,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/2985652bfabb423ab0d21d9ad5945cdd, entries=19, sequenceid=15, filesize=8.2 K 2023-02-21 09:57:32,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/.tmp/cf/8890a75f6e2c4a88984820bc67655763 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/8890a75f6e2c4a88984820bc67655763 2023-02-21 09:57:32,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.24 KB/1271, heapSize ~2.91 KB/2976, currentSize=0 B/0 for ca9f39730d0c524ed83cab2e18c641c8 in 37ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:32,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:32,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab. 2023-02-21 09:57:32,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6472b4dfac0a398ae332dd5f9936afab: 2023-02-21 09:57:32,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/8890a75f6e2c4a88984820bc67655763, entries=34, sequenceid=15, filesize=10.8 K 2023-02-21 09:57:32,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:32,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8. 2023-02-21 09:57:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ca9f39730d0c524ed83cab2e18c641c8: 2023-02-21 09:57:32,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for b51ce6c2c565e8d6b692d66c994428bb in 40ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:32,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:32,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:32,525 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=6472b4dfac0a398ae332dd5f9936afab, regionState=CLOSED 2023-02-21 09:57:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9bdca281384ff4be099bfeae0c42d2c6, disabling compactions & flushes 2023-02-21 09:57:32,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:32,525 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452525"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973452525"}]},"ts":"1676973452525"} 2023-02-21 09:57:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. after waiting 0 ms 2023-02-21 09:57:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:32,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9bdca281384ff4be099bfeae0c42d2c6 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-21 09:57:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 14df7364e6a8c8dc1107428e33ce230a, disabling compactions & flushes 2023-02-21 09:57:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. after waiting 0 ms 2023-02-21 09:57:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 14df7364e6a8c8dc1107428e33ce230a 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-21 09:57:32,526 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=ca9f39730d0c524ed83cab2e18c641c8, regionState=CLOSED 2023-02-21 09:57:32,526 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452526"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973452526"}]},"ts":"1676973452526"} 2023-02-21 09:57:32,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:32,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb. 2023-02-21 09:57:32,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b51ce6c2c565e8d6b692d66c994428bb: 2023-02-21 09:57:32,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=275 2023-02-21 09:57:32,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=275, state=SUCCESS; CloseRegionProcedure 6472b4dfac0a398ae332dd5f9936afab, server=jenkins-hbase4.apache.org,44835,1676973419584 in 197 msec 2023-02-21 09:57:32,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:32,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:32,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 81b4d0d7d383dbd3ed3b6da2e1fab12e, disabling compactions & flushes 2023-02-21 09:57:32,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:32,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:32,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. after waiting 0 ms 2023-02-21 09:57:32,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:32,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 81b4d0d7d383dbd3ed3b6da2e1fab12e 1/1 column families, dataSize=21.92 KB heapSize=47.36 KB 2023-02-21 09:57:32,536 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=b51ce6c2c565e8d6b692d66c994428bb, regionState=CLOSED 2023-02-21 09:57:32,536 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=274 2023-02-21 09:57:32,536 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973452536"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973452536"}]},"ts":"1676973452536"} 2023-02-21 09:57:32,536 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=274, state=SUCCESS; CloseRegionProcedure ca9f39730d0c524ed83cab2e18c641c8, server=jenkins-hbase4.apache.org,41011,1676973419614 in 197 msec 2023-02-21 09:57:32,536 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=6472b4dfac0a398ae332dd5f9936afab, UNASSIGN in 204 msec 2023-02-21 09:57:32,537 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=ca9f39730d0c524ed83cab2e18c641c8, UNASSIGN in 208 msec 2023-02-21 09:57:32,538 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-21 09:57:32,538 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure b51ce6c2c565e8d6b692d66c994428bb, server=jenkins-hbase4.apache.org,46077,1676973419539 in 203 msec 2023-02-21 09:57:32,539 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=b51ce6c2c565e8d6b692d66c994428bb, UNASSIGN in 210 msec 2023-02-21 09:57:32,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230221caf66a8391e84023aa82b1c4f4a28b71 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221caf66a8391e84023aa82b1c4f4a28b71 2023-02-21 09:57:32,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-21 09:57:32,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302216008504cf9b64e2593d548be4c267a87 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b202302216008504cf9b64e2593d548be4c267a87 2023-02-21 09:57:32,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302219e9c4cbddc034643a6c9a96380d5bf45 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d5202302219e9c4cbddc034643a6c9a96380d5bf45 2023-02-21 09:57:32,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/.tmp/cf/a6ad8979e2f6424d9a270ab5bfb86d59 2023-02-21 09:57:32,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/.tmp/cf/a6ad8979e2f6424d9a270ab5bfb86d59 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/a6ad8979e2f6424d9a270ab5bfb86d59 2023-02-21 09:57:32,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/.tmp/cf/613d44cea87c47f78e3b7b858b88838c 2023-02-21 09:57:32,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/a6ad8979e2f6424d9a270ab5bfb86d59, entries=39, sequenceid=15, filesize=11.6 K 2023-02-21 09:57:32,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/.tmp/cf/613d44cea87c47f78e3b7b858b88838c as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/613d44cea87c47f78e3b7b858b88838c 2023-02-21 09:57:32,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 14df7364e6a8c8dc1107428e33ce230a in 86ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:32,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/613d44cea87c47f78e3b7b858b88838c, entries=32, sequenceid=15, filesize=10.4 K 2023-02-21 09:57:32,615 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 9bdca281384ff4be099bfeae0c42d2c6 in 90ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:32,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:32,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a. 2023-02-21 09:57:32,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 14df7364e6a8c8dc1107428e33ce230a: 2023-02-21 09:57:32,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/.tmp/cf/fad7bb1ca857414aa1ec00a756cada2e 2023-02-21 09:57:32,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:32,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6. 2023-02-21 09:57:32,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9bdca281384ff4be099bfeae0c42d2c6: 2023-02-21 09:57:32,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:32,623 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=14df7364e6a8c8dc1107428e33ce230a, regionState=CLOSED 2023-02-21 09:57:32,623 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452623"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973452623"}]},"ts":"1676973452623"} 2023-02-21 09:57:32,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:32,623 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=9bdca281384ff4be099bfeae0c42d2c6, regionState=CLOSED 2023-02-21 09:57:32,623 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676973452623"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973452623"}]},"ts":"1676973452623"} 2023-02-21 09:57:32,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/.tmp/cf/fad7bb1ca857414aa1ec00a756cada2e as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/fad7bb1ca857414aa1ec00a756cada2e 2023-02-21 09:57:32,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=272 2023-02-21 09:57:32,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=272, state=SUCCESS; CloseRegionProcedure 14df7364e6a8c8dc1107428e33ce230a, server=jenkins-hbase4.apache.org,41011,1676973419614 in 293 msec 2023-02-21 09:57:32,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=273 2023-02-21 09:57:32,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=273, state=SUCCESS; CloseRegionProcedure 9bdca281384ff4be099bfeae0c42d2c6, server=jenkins-hbase4.apache.org,44835,1676973419584 in 293 msec 2023-02-21 09:57:32,626 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=14df7364e6a8c8dc1107428e33ce230a, UNASSIGN in 297 msec 2023-02-21 09:57:32,626 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=9bdca281384ff4be099bfeae0c42d2c6, UNASSIGN in 298 msec 2023-02-21 09:57:32,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/fad7bb1ca857414aa1ec00a756cada2e, entries=335, sequenceid=15, filesize=62.6 K 2023-02-21 09:57:32,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.92 KB/22441, heapSize ~47.34 KB/48480, currentSize=0 B/0 for 81b4d0d7d383dbd3ed3b6da2e1fab12e in 96ms, sequenceid=15, compaction requested=false 2023-02-21 09:57:32,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-21 09:57:32,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e. 2023-02-21 09:57:32,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 81b4d0d7d383dbd3ed3b6da2e1fab12e: 2023-02-21 09:57:32,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:32,633 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=81b4d0d7d383dbd3ed3b6da2e1fab12e, regionState=CLOSED 2023-02-21 09:57:32,633 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676973452633"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676973452633"}]},"ts":"1676973452633"} 2023-02-21 09:57:32,634 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=276 2023-02-21 09:57:32,635 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=276, state=SUCCESS; CloseRegionProcedure 81b4d0d7d383dbd3ed3b6da2e1fab12e, server=jenkins-hbase4.apache.org,46077,1676973419539 in 302 msec 2023-02-21 09:57:32,636 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=276, resume processing ppid=270 2023-02-21 09:57:32,636 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676973446507, region=81b4d0d7d383dbd3ed3b6da2e1fab12e, UNASSIGN in 307 msec 2023-02-21 09:57:32,637 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676973452637"}]},"ts":"1676973452637"} 2023-02-21 09:57:32,637 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676973446507, state=DISABLED in hbase:meta 2023-02-21 09:57:32,640 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676973446507 to state=DISABLED 2023-02-21 09:57:32,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676973446507 in 319 msec 2023-02-21 09:57:32,785 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-21 09:57:33,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-21 09:57:33,076 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 270 completed 2023-02-21 09:57:33,077 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,079 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,080 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-21 09:57:33,086 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:33,086 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:33,086 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:33,086 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:33,086 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:33,086 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:33,088 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits] 2023-02-21 09:57:33,089 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits] 2023-02-21 09:57:33,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits] 2023-02-21 09:57:33,089 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits] 2023-02-21 09:57:33,089 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits] 2023-02-21 09:57:33,089 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf, FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits] 2023-02-21 09:57:33,093 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/a2072e3f8aad440c865a4c001b6b7eb1 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/a2072e3f8aad440c865a4c001b6b7eb1 2023-02-21 09:57:33,094 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/8890a75f6e2c4a88984820bc67655763 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/8890a75f6e2c4a88984820bc67655763 2023-02-21 09:57:33,094 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/2985652bfabb423ab0d21d9ad5945cdd to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/2985652bfabb423ab0d21d9ad5945cdd 2023-02-21 09:57:33,094 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/a6ad8979e2f6424d9a270ab5bfb86d59 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/cf/a6ad8979e2f6424d9a270ab5bfb86d59 2023-02-21 09:57:33,094 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/f6a6cebecebc42b7920526e24f376d47 2023-02-21 09:57:33,094 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/1e36358cbb014a39be0ea2aee7b17873 2023-02-21 09:57:33,095 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/cf/eb37b03d9d2c471e9586d329a55957b9 2023-02-21 09:57:33,095 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/cf/8b29a5f1653247d09b1f72bfd86ddf1e 2023-02-21 09:57:33,096 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/cf/d06c68a1261349fdaea54214d1f6adfa 2023-02-21 09:57:33,096 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/fad7bb1ca857414aa1ec00a756cada2e to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/cf/fad7bb1ca857414aa1ec00a756cada2e 2023-02-21 09:57:33,096 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/613d44cea87c47f78e3b7b858b88838c to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/cf/613d44cea87c47f78e3b7b858b88838c 2023-02-21 09:57:33,098 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a/recovered.edits/18.seqid 2023-02-21 09:57:33,098 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/14df7364e6a8c8dc1107428e33ce230a 2023-02-21 09:57:33,099 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb/recovered.edits/18.seqid 2023-02-21 09:57:33,099 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8/recovered.edits/18.seqid 2023-02-21 09:57:33,099 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab/recovered.edits/18.seqid 2023-02-21 09:57:33,099 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e/recovered.edits/18.seqid 2023-02-21 09:57:33,100 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/b51ce6c2c565e8d6b692d66c994428bb 2023-02-21 09:57:33,100 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/ca9f39730d0c524ed83cab2e18c641c8 2023-02-21 09:57:33,100 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/6472b4dfac0a398ae332dd5f9936afab 2023-02-21 09:57:33,100 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/81b4d0d7d383dbd3ed3b6da2e1fab12e 2023-02-21 09:57:33,100 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits/18.seqid to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6/recovered.edits/18.seqid 2023-02-21 09:57:33,101 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/.tmp/data/default/testCorruptedSnapshot-1676973446507/9bdca281384ff4be099bfeae0c42d2c6 2023-02-21 09:57:33,101 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676973446507 regions 2023-02-21 09:57:33,101 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e 2023-02-21 09:57:33,101 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf] 2023-02-21 09:57:33,104 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c202302213db7cd893bd6460d89d9928822989130 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c202302213db7cd893bd6460d89d9928822989130 2023-02-21 09:57:33,105 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c2023022143c4a23bba144669a4959b54289d98ea to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/a87ff679a2f3e71d9181a67b7542122c2023022143c4a23bba144669a4959b54289d98ea 2023-02-21 09:57:33,106 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b202302216008504cf9b64e2593d548be4c267a87 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b202302216008504cf9b64e2593d548be4c267a87 2023-02-21 09:57:33,107 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b20230221d1b95928585f4161bd7f1160338f01eb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c4ca4238a0b923820dcc509a6f75849b20230221d1b95928585f4161bd7f1160338f01eb 2023-02-21 09:57:33,108 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221caf66a8391e84023aa82b1c4f4a28b71 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221caf66a8391e84023aa82b1c4f4a28b71 2023-02-21 09:57:33,109 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221df3103c8480844a18c55fe1d917da6f3 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/c81e728d9d4c2f636f067f89cc14862c20230221df3103c8480844a18c55fe1d917da6f3 2023-02-21 09:57:33,110 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e2023022116b5057424584fa4b1e93ea14181e870 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e2023022116b5057424584fa4b1e93ea14181e870 2023-02-21 09:57:33,110 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e202302211803b10023214dca9b99ba16f2b656cb to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/d41d8cd98f00b204e9800998ecf8427e202302211803b10023214dca9b99ba16f2b656cb 2023-02-21 09:57:33,111 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d5202302219e9c4cbddc034643a6c9a96380d5bf45 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d5202302219e9c4cbddc034643a6c9a96380d5bf45 2023-02-21 09:57:33,112 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d520230221afee0ccb4b734defb1ba85583706faec to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/e4da3b7fbbce2345d7772b0674a318d520230221afee0ccb4b734defb1ba85583706faec 2023-02-21 09:57:33,113 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221e7829692f3e44783ad2e74eb248fcf41 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221e7829692f3e44783ad2e74eb248fcf41 2023-02-21 09:57:33,114 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221fb90d0632ed14a6cac03f10e01135c01 to hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/archive/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230221fb90d0632ed14a6cac03f10e01135c01 2023-02-21 09:57:33,114 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/mobdir/data/default/testCorruptedSnapshot-1676973446507/63964adaebfd4cae5d410cfae8f6824e 2023-02-21 09:57:33,116 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,118 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676973446507 from hbase:meta 2023-02-21 09:57:33,119 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676973446507' descriptor. 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676973446507' from region states. 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973453120"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973453120"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973453120"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973453120"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973453120"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676973453120"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,122 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-21 09:57:33,122 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b51ce6c2c565e8d6b692d66c994428bb, NAME => 'testCorruptedSnapshot-1676973446507,,1676973446509.b51ce6c2c565e8d6b692d66c994428bb.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 14df7364e6a8c8dc1107428e33ce230a, NAME => 'testCorruptedSnapshot-1676973446507,1,1676973446509.14df7364e6a8c8dc1107428e33ce230a.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 9bdca281384ff4be099bfeae0c42d2c6, NAME => 'testCorruptedSnapshot-1676973446507,2,1676973446509.9bdca281384ff4be099bfeae0c42d2c6.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => ca9f39730d0c524ed83cab2e18c641c8, NAME => 'testCorruptedSnapshot-1676973446507,3,1676973446509.ca9f39730d0c524ed83cab2e18c641c8.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 6472b4dfac0a398ae332dd5f9936afab, NAME => 'testCorruptedSnapshot-1676973446507,4,1676973446509.6472b4dfac0a398ae332dd5f9936afab.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 81b4d0d7d383dbd3ed3b6da2e1fab12e, NAME => 'testCorruptedSnapshot-1676973446507,5,1676973446509.81b4d0d7d383dbd3ed3b6da2e1fab12e.', STARTKEY => '5', ENDKEY => ''}] 2023-02-21 09:57:33,122 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676973446507' as deleted. 2023-02-21 09:57:33,122 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676973446507","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676973453122"}]},"ts":"9223372036854775807"} 2023-02-21 09:57:33,123 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676973446507 state from META 2023-02-21 09:57:33,125 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 2023-02-21 09:57:33,125 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676973446507 in 48 msec 2023-02-21 09:57:33,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-21 09:57:33,331 INFO [Listener at localhost/40243] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676973446507, procId: 283 completed 2023-02-21 09:57:33,335 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676973446507" 2023-02-21 09:57:33,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676973446507 2023-02-21 09:57:33,338 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676973446507" 2023-02-21 09:57:33,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41687] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676973446507 2023-02-21 09:57:33,358 INFO [Listener at localhost/40243] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=420 (was 424), OpenFileDescriptor=795 (was 814), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 249), ProcessCount=173 (was 172) - ProcessCount LEAK? -, AvailableMemoryMB=5196 (was 5229) 2023-02-21 09:57:33,358 INFO [Listener at localhost/40243] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-21 09:57:33,359 INFO [Listener at localhost/40243] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-21 09:57:33,360 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c633018 to 127.0.0.1:58856 2023-02-21 09:57:33,360 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,360 DEBUG [Listener at localhost/40243] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-21 09:57:33,360 DEBUG [Listener at localhost/40243] util.JVMClusterUtil(257): Found active master hash=1375978211, stopped=false 2023-02-21 09:57:33,360 INFO [Listener at localhost/40243] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:57:33,364 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:33,364 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:33,364 INFO [Listener at localhost/40243] procedure2.ProcedureExecutor(629): Stopping 2023-02-21 09:57:33,364 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:33,364 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-21 09:57:33,364 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:33,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:57:33,365 DEBUG [Listener at localhost/40243] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f853f68 to 127.0.0.1:58856 2023-02-21 09:57:33,365 DEBUG [Listener at localhost/40243] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,365 INFO [Listener at localhost/40243] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46077,1676973419539' ***** 2023-02-21 09:57:33,365 INFO [Listener at localhost/40243] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-21 09:57:33,365 INFO [Listener at localhost/40243] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44835,1676973419584' ***** 2023-02-21 09:57:33,365 INFO [Listener at localhost/40243] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-21 09:57:33,365 INFO [Listener at localhost/40243] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41011,1676973419614' ***** 2023-02-21 09:57:33,365 INFO [Listener at localhost/40243] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-21 09:57:33,365 INFO [RS:1;jenkins-hbase4:44835] regionserver.HeapMemoryManager(220): Stopping 2023-02-21 09:57:33,365 INFO [RS:2;jenkins-hbase4:41011] regionserver.HeapMemoryManager(220): Stopping 2023-02-21 09:57:33,365 INFO [RS:0;jenkins-hbase4:46077] regionserver.HeapMemoryManager(220): Stopping 2023-02-21 09:57:33,366 INFO [RS:1;jenkins-hbase4:44835] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-21 09:57:33,366 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-21 09:57:33,366 INFO [RS:1;jenkins-hbase4:44835] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-21 09:57:33,366 INFO [RS:0;jenkins-hbase4:46077] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-21 09:57:33,366 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:33,366 INFO [RS:2;jenkins-hbase4:41011] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-21 09:57:33,366 DEBUG [RS:1;jenkins-hbase4:44835] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2556f54f to 127.0.0.1:58856 2023-02-21 09:57:33,366 INFO [RS:0;jenkins-hbase4:46077] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-21 09:57:33,366 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-21 09:57:33,366 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-21 09:57:33,366 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:33,366 DEBUG [RS:1;jenkins-hbase4:44835] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,366 INFO [RS:2;jenkins-hbase4:41011] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-21 09:57:33,366 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44835,1676973419584; all regions closed. 2023-02-21 09:57:33,366 DEBUG [RS:0;jenkins-hbase4:46077] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5cf5f6aa to 127.0.0.1:58856 2023-02-21 09:57:33,366 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(3304): Received CLOSE for 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:33,366 DEBUG [RS:0;jenkins-hbase4:46077] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,367 INFO [RS:0;jenkins-hbase4:46077] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-21 09:57:33,367 INFO [RS:0;jenkins-hbase4:46077] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-21 09:57:33,367 INFO [RS:0;jenkins-hbase4:46077] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-21 09:57:33,367 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-21 09:57:33,367 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:33,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:57:33,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 46e4fb13fce46594ac37583388eb6688, disabling compactions & flushes 2023-02-21 09:57:33,367 DEBUG [RS:2;jenkins-hbase4:41011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e195c33 to 127.0.0.1:58856 2023-02-21 09:57:33,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:57:33,367 DEBUG [RS:2;jenkins-hbase4:41011] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:33,367 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-21 09:57:33,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:33,367 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1479): Online Regions={46e4fb13fce46594ac37583388eb6688=hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688.} 2023-02-21 09:57:33,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-21 09:57:33,367 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-21 09:57:33,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. after waiting 0 ms 2023-02-21 09:57:33,368 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-21 09:57:33,367 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-21 09:57:33,368 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-21 09:57:33,368 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-21 09:57:33,369 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-21 09:57:33,369 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-21 09:57:33,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:33,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-21 09:57:33,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 46e4fb13fce46594ac37583388eb6688 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-21 09:57:33,369 DEBUG [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1505): Waiting on 46e4fb13fce46594ac37583388eb6688 2023-02-21 09:57:33,369 DEBUG [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-21 09:57:33,376 DEBUG [RS:1;jenkins-hbase4:44835] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs 2023-02-21 09:57:33,376 INFO [RS:1;jenkins-hbase4:44835] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44835%2C1676973419584:(num 1676973421119) 2023-02-21 09:57:33,376 DEBUG [RS:1;jenkins-hbase4:44835] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,376 INFO [RS:1;jenkins-hbase4:44835] regionserver.LeaseManager(133): Closed leases 2023-02-21 09:57:33,376 INFO [RS:1;jenkins-hbase4:44835] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-21 09:57:33,377 INFO [RS:1;jenkins-hbase4:44835] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-21 09:57:33,377 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-21 09:57:33,377 INFO [RS:1;jenkins-hbase4:44835] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-21 09:57:33,377 INFO [RS:1;jenkins-hbase4:44835] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-21 09:57:33,378 INFO [RS:1;jenkins-hbase4:44835] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44835 2023-02-21 09:57:33,385 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:33,385 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:33,385 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:33,385 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:33,385 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44835,1676973419584 2023-02-21 09:57:33,385 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:33,389 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@13ab6691 rejected from java.util.concurrent.ThreadPoolExecutor@1794e03c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,389 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:33,390 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b59bb5f rejected from java.util.concurrent.ThreadPoolExecutor@1794e03c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/.tmp/info/843e1ccc8822430aa4b14f8305239c8f 2023-02-21 09:57:33,390 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44835,1676973419584] 2023-02-21 09:57:33,391 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44835,1676973419584; numProcessing=1 2023-02-21 09:57:33,393 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44835,1676973419584 already deleted, retry=false 2023-02-21 09:57:33,393 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44835,1676973419584 expired; onlineServers=2 2023-02-21 09:57:33,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/.tmp/info/843e1ccc8822430aa4b14f8305239c8f as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/info/843e1ccc8822430aa4b14f8305239c8f 2023-02-21 09:57:33,396 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/.tmp/info/bd36639bac67476da8ff01a2f89284ea 2023-02-21 09:57:33,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/info/843e1ccc8822430aa4b14f8305239c8f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-21 09:57:33,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 46e4fb13fce46594ac37583388eb6688 in 29ms, sequenceid=6, compaction requested=false 2023-02-21 09:57:33,400 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd36639bac67476da8ff01a2f89284ea 2023-02-21 09:57:33,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/namespace/46e4fb13fce46594ac37583388eb6688/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-21 09:57:33,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:33,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 46e4fb13fce46594ac37583388eb6688: 2023-02-21 09:57:33,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676973421501.46e4fb13fce46594ac37583388eb6688. 2023-02-21 09:57:33,410 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/.tmp/rep_barrier/3b7aacb9d3004b8ca0caacf82baf0fc9 2023-02-21 09:57:33,413 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3b7aacb9d3004b8ca0caacf82baf0fc9 2023-02-21 09:57:33,421 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/.tmp/table/178fca25b4754ed182b6ac4c0e460ec6 2023-02-21 09:57:33,422 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-21 09:57:33,422 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-21 09:57:33,422 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-21 09:57:33,424 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 178fca25b4754ed182b6ac4c0e460ec6 2023-02-21 09:57:33,424 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/.tmp/info/bd36639bac67476da8ff01a2f89284ea as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/info/bd36639bac67476da8ff01a2f89284ea 2023-02-21 09:57:33,427 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd36639bac67476da8ff01a2f89284ea 2023-02-21 09:57:33,427 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/info/bd36639bac67476da8ff01a2f89284ea, entries=64, sequenceid=307, filesize=11.8 K 2023-02-21 09:57:33,427 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/.tmp/rep_barrier/3b7aacb9d3004b8ca0caacf82baf0fc9 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/rep_barrier/3b7aacb9d3004b8ca0caacf82baf0fc9 2023-02-21 09:57:33,430 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3b7aacb9d3004b8ca0caacf82baf0fc9 2023-02-21 09:57:33,430 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/rep_barrier/3b7aacb9d3004b8ca0caacf82baf0fc9, entries=18, sequenceid=307, filesize=7.0 K 2023-02-21 09:57:33,431 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/.tmp/table/178fca25b4754ed182b6ac4c0e460ec6 as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/table/178fca25b4754ed182b6ac4c0e460ec6 2023-02-21 09:57:33,433 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 178fca25b4754ed182b6ac4c0e460ec6 2023-02-21 09:57:33,433 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/table/178fca25b4754ed182b6ac4c0e460ec6, entries=22, sequenceid=307, filesize=7.0 K 2023-02-21 09:57:33,434 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 64ms, sequenceid=307, compaction requested=false 2023-02-21 09:57:33,437 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-21 09:57:33,437 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-21 09:57:33,439 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-21 09:57:33,439 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-21 09:57:33,439 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-21 09:57:33,563 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:33,563 INFO [RS:1;jenkins-hbase4:44835] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44835,1676973419584; zookeeper connection closed. 2023-02-21 09:57:33,563 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@53ab5606 rejected from java.util.concurrent.ThreadPoolExecutor@1794e03c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,563 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:44835-0x100764f8e080002, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:33,563 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@35c98fe4] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@35c98fe4 2023-02-21 09:57:33,563 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@545c45ae rejected from java.util.concurrent.ThreadPoolExecutor@1794e03c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,569 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41011,1676973419614; all regions closed. 2023-02-21 09:57:33,569 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46077,1676973419539; all regions closed. 2023-02-21 09:57:33,576 DEBUG [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs 2023-02-21 09:57:33,576 DEBUG [RS:2;jenkins-hbase4:41011] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs 2023-02-21 09:57:33,576 INFO [RS:2;jenkins-hbase4:41011] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41011%2C1676973419614:(num 1676973421119) 2023-02-21 09:57:33,576 DEBUG [RS:2;jenkins-hbase4:41011] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,576 INFO [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46077%2C1676973419539.meta:.meta(num 1676973421237) 2023-02-21 09:57:33,576 INFO [RS:2;jenkins-hbase4:41011] regionserver.LeaseManager(133): Closed leases 2023-02-21 09:57:33,577 INFO [RS:2;jenkins-hbase4:41011] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-21 09:57:33,577 INFO [RS:2;jenkins-hbase4:41011] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-21 09:57:33,577 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-21 09:57:33,577 INFO [RS:2;jenkins-hbase4:41011] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-21 09:57:33,577 INFO [RS:2;jenkins-hbase4:41011] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-21 09:57:33,578 INFO [RS:2;jenkins-hbase4:41011] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41011 2023-02-21 09:57:33,585 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:33,585 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:33,585 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41011,1676973419614 2023-02-21 09:57:33,585 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f496971 rejected from java.util.concurrent.ThreadPoolExecutor@b76ec40[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,586 DEBUG [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/oldWALs 2023-02-21 09:57:33,586 INFO [RS:0;jenkins-hbase4:46077] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46077%2C1676973419539:(num 1676973421119) 2023-02-21 09:57:33,586 DEBUG [RS:0;jenkins-hbase4:46077] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,586 INFO [RS:0;jenkins-hbase4:46077] regionserver.LeaseManager(133): Closed leases 2023-02-21 09:57:33,586 INFO [RS:0;jenkins-hbase4:46077] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-21 09:57:33,587 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-21 09:57:33,587 INFO [RS:0;jenkins-hbase4:46077] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46077 2023-02-21 09:57:33,587 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41011,1676973419614] 2023-02-21 09:57:33,587 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41011,1676973419614; numProcessing=2 2023-02-21 09:57:33,687 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:33,687 INFO [RS:2;jenkins-hbase4:41011] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41011,1676973419614; zookeeper connection closed. 2023-02-21 09:57:33,688 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b36dab3 rejected from java.util.concurrent.ThreadPoolExecutor@b76ec40[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,688 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:41011-0x100764f8e080003, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:33,688 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43d8c907 rejected from java.util.concurrent.ThreadPoolExecutor@b76ec40[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,688 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6ee70ec8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6ee70ec8 2023-02-21 09:57:33,689 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41011,1676973419614 already deleted, retry=false 2023-02-21 09:57:33,689 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41011,1676973419614 expired; onlineServers=1 2023-02-21 09:57:33,689 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46077,1676973419539 2023-02-21 09:57:33,689 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-21 09:57:33,689 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14daa2f9 rejected from java.util.concurrent.ThreadPoolExecutor@10e524ba[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:33,691 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46077,1676973419539] 2023-02-21 09:57:33,692 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46077,1676973419539; numProcessing=3 2023-02-21 09:57:33,693 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46077,1676973419539 already deleted, retry=false 2023-02-21 09:57:33,693 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46077,1676973419539 expired; onlineServers=0 2023-02-21 09:57:33,693 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41687,1676973418380' ***** 2023-02-21 09:57:33,693 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-21 09:57:33,694 DEBUG [M:0;jenkins-hbase4:41687] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@133ec8d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-21 09:57:33,694 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41687,1676973418380 2023-02-21 09:57:33,694 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41687,1676973418380; all regions closed. 2023-02-21 09:57:33,694 DEBUG [M:0;jenkins-hbase4:41687] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-21 09:57:33,694 DEBUG [M:0;jenkins-hbase4:41687] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-21 09:57:33,694 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-21 09:57:33,694 DEBUG [M:0;jenkins-hbase4:41687] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-21 09:57:33,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676973420752] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676973420752,5,FailOnTimeoutGroup] 2023-02-21 09:57:33,694 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676973420752] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676973420752,5,FailOnTimeoutGroup] 2023-02-21 09:57:33,694 INFO [M:0;jenkins-hbase4:41687] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-21 09:57:33,694 INFO [M:0;jenkins-hbase4:41687] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-21 09:57:33,695 INFO [M:0;jenkins-hbase4:41687] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-21 09:57:33,695 DEBUG [M:0;jenkins-hbase4:41687] master.HMaster(1512): Stopping service threads 2023-02-21 09:57:33,695 INFO [M:0;jenkins-hbase4:41687] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-21 09:57:33,696 ERROR [M:0;jenkins-hbase4:41687] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-21 09:57:33,696 INFO [M:0;jenkins-hbase4:41687] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-21 09:57:33,696 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-21 09:57:33,696 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-21 09:57:33,696 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-21 09:57:33,697 DEBUG [M:0;jenkins-hbase4:41687] zookeeper.ZKUtil(398): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-21 09:57:33,697 WARN [M:0;jenkins-hbase4:41687] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-21 09:57:33,697 INFO [M:0;jenkins-hbase4:41687] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-21 09:57:33,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-21 09:57:33,697 INFO [M:0;jenkins-hbase4:41687] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-21 09:57:33,697 DEBUG [M:0;jenkins-hbase4:41687] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-21 09:57:33,697 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:33,697 DEBUG [M:0;jenkins-hbase4:41687] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:33,697 DEBUG [M:0;jenkins-hbase4:41687] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-21 09:57:33,697 DEBUG [M:0;jenkins-hbase4:41687] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:33,697 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.54 KB heapSize=1.01 MB 2023-02-21 09:57:33,712 INFO [M:0;jenkins-hbase4:41687] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.54 KB at sequenceid=2109 (bloomFilter=true), to=hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/08e46c24c6cd4e27833d7c0cb91c236a 2023-02-21 09:57:33,715 INFO [M:0;jenkins-hbase4:41687] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08e46c24c6cd4e27833d7c0cb91c236a 2023-02-21 09:57:33,715 DEBUG [M:0;jenkins-hbase4:41687] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/08e46c24c6cd4e27833d7c0cb91c236a as hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/08e46c24c6cd4e27833d7c0cb91c236a 2023-02-21 09:57:33,718 INFO [M:0;jenkins-hbase4:41687] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08e46c24c6cd4e27833d7c0cb91c236a 2023-02-21 09:57:33,718 INFO [M:0;jenkins-hbase4:41687] regionserver.HStore(1080): Added hdfs://localhost:35503/user/jenkins/test-data/53303bfb-46d7-a9db-aff3-fc4931b34926/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/08e46c24c6cd4e27833d7c0cb91c236a, entries=282, sequenceid=2109, filesize=30.9 K 2023-02-21 09:57:33,719 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegion(2947): Finished flush of dataSize ~845.54 KB/865837, heapSize ~1.01 MB/1061872, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=2109, compaction requested=false 2023-02-21 09:57:33,719 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-21 09:57:33,719 DEBUG [M:0;jenkins-hbase4:41687] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-21 09:57:33,724 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-21 09:57:33,724 INFO [M:0;jenkins-hbase4:41687] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-21 09:57:33,724 INFO [M:0;jenkins-hbase4:41687] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41687 2023-02-21 09:57:33,726 DEBUG [M:0;jenkins-hbase4:41687] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,41687,1676973418380 already deleted, retry=false 2023-02-21 09:57:34,064 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:34,064 INFO [M:0;jenkins-hbase4:41687] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41687,1676973418380; zookeeper connection closed. 2023-02-21 09:57:34,064 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@44d39f93 rejected from java.util.concurrent.ThreadPoolExecutor@3f5c750[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:34,064 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): master:41687-0x100764f8e080000, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:34,064 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6de45497 rejected from java.util.concurrent.ThreadPoolExecutor@3f5c750[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:34,164 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:34,164 INFO [RS:0;jenkins-hbase4:46077] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46077,1676973419539; zookeeper connection closed. 2023-02-21 09:57:34,164 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@25f4deab rejected from java.util.concurrent.ThreadPoolExecutor@10e524ba[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:34,164 DEBUG [Listener at localhost/40243-EventThread] zookeeper.ZKWatcher(600): regionserver:46077-0x100764f8e080001, quorum=127.0.0.1:58856, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-21 09:57:34,165 ERROR [Listener at localhost/40243-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d015557 rejected from java.util.concurrent.ThreadPoolExecutor@10e524ba[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-21 09:57:34,165 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1dc6343e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1dc6343e 2023-02-21 09:57:34,165 INFO [Listener at localhost/40243] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-21 09:57:34,165 WARN [Listener at localhost/40243] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-21 09:57:34,168 INFO [Listener at localhost/40243] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-21 09:57:34,273 WARN [BP-1991167450-172.31.14.131-1676973415272 heartbeating to localhost/127.0.0.1:35503] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-21 09:57:34,273 WARN [BP-1991167450-172.31.14.131-1676973415272 heartbeating to localhost/127.0.0.1:35503] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1991167450-172.31.14.131-1676973415272 (Datanode Uuid d9b45635-f12f-447d-8966-6f56b17879d6) service to localhost/127.0.0.1:35503 2023-02-21 09:57:34,275 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data5/current/BP-1991167450-172.31.14.131-1676973415272] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-21 09:57:34,275 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data6/current/BP-1991167450-172.31.14.131-1676973415272] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-21 09:57:34,276 WARN [Listener at localhost/40243] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-21 09:57:34,278 INFO [Listener at localhost/40243] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-21 09:57:34,383 WARN [BP-1991167450-172.31.14.131-1676973415272 heartbeating to localhost/127.0.0.1:35503] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-21 09:57:34,383 WARN [BP-1991167450-172.31.14.131-1676973415272 heartbeating to localhost/127.0.0.1:35503] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1991167450-172.31.14.131-1676973415272 (Datanode Uuid a8510abe-d1ee-4d75-bfe9-cb5544285d6e) service to localhost/127.0.0.1:35503 2023-02-21 09:57:34,384 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data3/current/BP-1991167450-172.31.14.131-1676973415272] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-21 09:57:34,384 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data4/current/BP-1991167450-172.31.14.131-1676973415272] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-21 09:57:34,385 WARN [Listener at localhost/40243] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-21 09:57:34,391 INFO [Listener at localhost/40243] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-21 09:57:34,497 WARN [BP-1991167450-172.31.14.131-1676973415272 heartbeating to localhost/127.0.0.1:35503] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-21 09:57:34,497 WARN [BP-1991167450-172.31.14.131-1676973415272 heartbeating to localhost/127.0.0.1:35503] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1991167450-172.31.14.131-1676973415272 (Datanode Uuid a7765cd4-64e4-40e6-b68d-4e6de46e2658) service to localhost/127.0.0.1:35503 2023-02-21 09:57:34,498 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data1/current/BP-1991167450-172.31.14.131-1676973415272] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-21 09:57:34,498 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/af683303-bfd5-8ff0-f08c-30a8010b447f/cluster_69142b59-50ea-f10f-dae7-f7105a9c7a88/dfs/data/data2/current/BP-1991167450-172.31.14.131-1676973415272] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-21 09:57:34,528 INFO [Listener at localhost/40243] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-21 09:57:34,660 INFO [Listener at localhost/40243] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-21 09:57:34,749 INFO [Listener at localhost/40243] hbase.HBaseTestingUtility(1293): Minicluster is down